TESSELLATIONS OF HOMOGENEOUS SPACES 
OF CLASSICAL GROUPS OF REAL RANK TWO 

ALESSANDRA lOZZI AND DAVE WITTE 

Abstract. Let H he a, closed, connected subgroup of a connected, simple Lie group G with 
finite center. The homogeneous space G/H has a tessellation if there is a discrete subgroup 
r of G, such that T acts properly discontinuously on G/H, and the double-coset space 
T\G/H is compact. Note that if either H or G/H is compact, then G/H has a tessellation; 
these are the obvious examples. 

It is not difficult to see that if G has real rank one, then only the obvious homogeneous 
spaces have tessellations. Thus, the first interesting case is when G has real rank two. In 
particular, R. Kulkarni and T. Kobayashi constructed examples that are not obvious when 
G — S0(2, 2n)° or SU(2, 2n). H. Oh and D. Witte constructed additional examples in both 
of these cases, and obtained a complete classification when G = S0(2, 2n)°. We simplify the 
work of Oh- Witte, and extend it to obtain a complete classification when G = SU(2,2ri). 
This includes the construction of another family of examples. 

The main results are obtained from methods of Y. Benoist and T. Kobayashi: we fix a 
Cartan decomposition G — KA~^K, and study the intersection [KHK) n . Our expo- 
sition generally assumes only the standard theory of connected Lie groups, although basic 
properties of real algebraic groups are sometimes also employed; the specialized techniques 
that we use are developed from a fairly elementary level. 
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1. Introduction 

1.1. Definition [KN, pp. 43-44]. A group F of homeomorphisms of a topological space M 
acts properly dis continuously on M if, for every compact subset C of M, 

{7eF|Cn7C7^0}is finite. 

Classically, a discrete group F of isometrics of a Riemannian manifold M is a crystallo- 
graphic group if F acts properly discontinuously on M, and the quotient F\M is compact. 
The F-translates of any fundamental domain for F\M form a tessellation of M. 

These notions generalize to any homogeneous space, even without an invariant metric. 

1.2. Definition. Let 

• G be a Lie group and 

• if be a closed subgroup of G. 

A discrete subgroup F of G is a crystallographic group for G/H if 

1) F acts properly discontinuously on G/H] and 

2) V\G/H is compact. 

We say that G/H has a tessellation if there exists a crystallographic group F for G/H. 

Crystallographic groups and the corresponding tessellations have been studied for many 
groups G. (A brief recent introduction to the subject is given in [Kb7].) The classical 
Bieberbach Theorems [Cha, Chap. 1] deal with the case where G is the group of isometrics 
of Euclidean space M" = G/H . As another example, the Auslander Conjecture [Abe, AMS, 
FG, Mrl, Tom] asserts that if G is the group of all affine transformations of M", then every 
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crystallographic group has a solvable subgroup of finite index. In addition, the case where 
G is solvable has been discussed in [Wit] . 

In this paper, we focus on the case where G is a simple Lie group, such as SL(n, M), 
SO(m, n), or SU(m, n). 

1.3. Standing assumptions. Throughout this paper: 

1) G is a linear, semisimple Lie group with only finitely many connected components; and 

2) is a closed subgroup of G with only finitely many connected components. 

lA. Remark. Because H/H° is finite (hence compact), it is easy to see that G/H has a 
tessellation if and only if G/H° has a tessellation. Also, if G/H has a tessellation, then 
G°/H° has a tessellation. Furthermore, the converse holds in many situations. (See §2D for 
a discussion of this issue.) Thus, there is usually no harm in assuming that both G and H are 
connected; we will feel free to do so whenever it is convenient. On the other hand, because 
SO(m, n) is usually not connected (it usually has two components [Hel, Lem. 10.2.4, p. 451]), 
it would be somewhat awkward to make this a blanket assumption. 

1.5. Example. There are two classical cases in which G/H is well known to have a tessel- 
lation. 

1) If G/H is compact, then we may let F = e (or any finite subgroup of G). 

2) If H is compact, then we may let F be any cocompact lattice in G. (A. Borel [Brl] 
proved that every connected, simple Lie group has a cocompact lattice.) 

Thus, the existence of a tessellation is an interesting question only when neither H nor G/H 
is compact. (In this case, any crystallographic group F must be infinite, and cannot be a 
lattice in G.) 

Given G (satisfying 1.3(1)), we would like to find all the subgroups H (satisfying 1.3(2)), 
such that G/H has a tessellation. This seems to be a difficult problem in general. (See the 
surveys [Kb5] and [Lab] for a discussion of the many partial results that have been obtained, 
mainly under the additional assumption that H is reductive.) However, it can be solved in 
certain cases of low real rank. In particular, as we will now briefiy explain, the problem is 
very easy if R-rank G = or 1 . Most of this paper is devoted to solving the problem for 
certain cases where M-rankG = 2. 

If R-rankG = (that is, if G is compact), then G/H must be compact (and H must also 
be compact), so G/H has a tessellation, but this is not interesting. If R-rankG = 1, then 
there arc some interesting homogeneous spaces, but it turns out that none of them have 
tessellations. 

1.6. Example. G = SL(2, R) is transitive on R^ — {0}, so R^ — {0} is a homogeneous space 
for G. It does not have a tessellation, for reasons that we now explain. 

Let G be the unit circle, so G is a compact subset of R^ — {0}. 

We claim that G fl gG ^ 0, for every g E G (cf. Figure 1.1). To see this, note that, 
because dei g = 1, the ellipse bounded by gC has the same area as the disk bounded by G, 
so gG cannot be contained in the interior of the disk bounded by G, and cannot contain G 
in its interior. Thus, gC must be partly inside G and partly outside, so gG must cross G, as 
claimed. 

Let F be any discrete subgroup of G. The preceding paragraph implies that G fl 7G 7^ 0, 
for every 7 e F. If F acts properly discontinuously on R^ — {0}, then, because G is compact. 
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Figure 1.1. The Calabi-Markus Phenomenon (Example 1.6): C fl gfC 7^ 0, 
for every g G SL(2,M), so no infinite subgroup of SL(2,M) acts properly dis- 
continuously. 

this implies that F is finite. So the quotient F\(M^ — {0}) is not compact. Therefore F is 
not a crystallographic group. We have shown that no subgroup of G is a crystallographic 
group, so we conclude that — {0} does not have a tessellation. 

This example illustrates the Calabi-Markus Phenomenon: if there is a compact subset C 
of G/H, such that CdgC 7^ 0, for every g & G, then no infinite subgroup of G acts properly 
discontinuously on G/H (see 2.8). Thus, G/H does not have a tessellation, unless G/H is 
compact (see 2.9). 

We will see in Section 2 that the following proposition can be proved quite easily from 
basic properties of the Cartan projection. 

2.16'. Proposition (cf. [Kb3, Lem. 3.2]). //M-rankG = 1, and H is not compact, then 
there is a compact subset C of G/H , such that G fl gG 7^ 0, for every g & G. 

1.7. Corollary (Kulkarni). //R-rankG = 1, and neither H nor G/H is compact, then 
G/H does not have a tessellation. 

We now consider groups of real rank two. The obvious example is SL(3,M), but, in this 
case, once again, none of the interesting homogeneous spaces have tessellations. Moreover, 
the same is true when real numbers are replaced by complex numbers or quaternions. The 
case where dim if = 1 relies on beautiful methods of Y. Benoist and F. Labourie [BL] or 
G. A. Margulis [Mr2], which we describe in Section 5. 

1.8. Theorem (Benoist, Margulis, Oh-Witte, see §6). // 

G = SL(3, R), SL(3, C), or SL(3, M), 
and neither H nor G/H is compact, then G/H does not have a tessellation. 

It is important to note that some interesting homogeneous spaces do have tessellations. 
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1.9. Example. Suppose G — L x H, and let F be a cocompact lattice in L. Then F acts 
properly discontinuously on L = G/H, and r\G/H = F\L is compact. So G/H has a 
tessellation. 

The following easy lemma generalizes this example to the situation where G is a more 
general product of L and H, not necessarily a direct product. 

9.1'. Lemma. Let H and L he closed subgroups of G, such that 

• G = LH, 

• L n H is com,pact; and 

• L has a cocompact lattice F. 

Then G/H has a tessellation. {Namely, T is a crystallographic group for G/H .) 

For G = S0(2,n) or SU(2,n), this lemma leads to some interesting examples found by 
R. Kulkarni [Kul, Thm. 6.1] and T. Kobayashi [Kbl, Prop. 4.9]. 

1.10. Example (Kulkarni, Kobayashi). There are natural embeddings 

S0(1, n) ^ S0(2, n) and SU(1, n) ^ SU(2, n). 

Furthermore, identifying C^+"^ with ]R2+2m yjpifjg embedding 

SU(l,m) ^ SO(2,2m). 

Similarly, identifying H^"*"™ with yields an embedding 

Sp(l,m) ^ SU(2,2m). 

Thus, we may think of S0(l,2m) and SU(l,m) as subgroups of SO(2,2m); and we may 
think of SU(l,2m) and Sp(l,m) as subgroups of SU(2,2m). 

With the above understanding, we see that S0(l,2m) is the stabilizer of a vector of 
norm +1. Since SU(1, m) is transitive on the set of such vectors, we have 

SO(2,2m) = S0(l,2m) SU(l,m). 

Similarly, 

SU(2, 2m) = SU(1, 2m) Sp(l, m). 

Then Lemma 9.1' implies that each of the following four homogeneous spaces has a tessella- 
tion: 

• SO(2,2m)/SO(l,2m), 

• SO(2,2m)/SU(l,m), 

• SU(2,2m)/SU(l,2m), and 

• SU(2,2m)/Sp(l,m). 

1.11. Remark. When discussing S0(2,n) or SU(2,n), we always assume n > 2. This causes 
no harm, because SO (2, 2) is locally isomorphic to SL(2, M) x SL(2, M) [Hel, (x), p. 520], and 
SU(2,2) is locally isomorphic to SO(2,4) [Hel, (vi), p. 519]. 

When n is even, H. Oh and D. Witte [0W3] provided a complete description of all the 
(closed, connected) subgroups H, such that SO (2, n)/if has a tessellation, but their classi- 
fication is not quite complete when n is odd. 

In this paper, we extend the work of Oh and Witte to obtain analogous results for homo- 
geneous spaces of G = SU(2,n). We also give a much shorter proof of the main results of 
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[0W3] . The same techniques should yield significant results for homogeneous spaces of the 
other simple groups of real rank two, although the calculations seem to be difficult. On the 
other hand, the groups of higher real rank require different ideas. 

Once one knows that a tessellation of G/H exists, it would be interesting to find all 
of the crystallographic groups for G/H and, for each crystallographic group, describe the 
possible tessellations. These are much more delicate questions, which we do not address 
at all. (W. Goldman [Gol], F. Salein [Sal], T. Kobayashi [Kb6], and A. Zeghib [Zeg] have 
interesting results in some special cases.) 

In the remainder of this introduction, we state the specific results for homogeneous spaces 
of S0(2,n) and SU(2,n). 

1.12. Notation (Iwasawa [Iwa, p. 533]). For any connected Lie group H, let 

d{H) = dimH -dim Kh, 

where Kh is any maximal compact subgroup of H. This is well defined, because all the 
maximal compact subgroups of H are conjugate [Hc2, Thm. 15.3.1(iii), pp. 180-181]. 

1.13. Example. If H is semisimple, we have the Iwasawa decomposition H = KhAhNh 
[Hel, Thm. 6.5.1, pp. 270-271], from which it is obvious that d{H) = dim(/l//A^j^). 

This yields the following calculations (see 7.5 and 7.15)): 

• (i(SO(l,n)) =n. 

• c^(S0(2,n)) = 2n. 

• rf(SU(l,n)) = 2n. 

• rf(SU(2,n)) = 4n. 

• d(Sp(l,n)) = in. 

1.14. Remark. If if C AN (for some Iwasawa decomposition G — KAN of G), then d{H) — 
dimH (see 3.18 and 3.15(3)). 

1.15. Notation [Kb4, Defn. 2.1.1]. For subgroups Hi and H2 of G, we write Hi ~ H2 if 
there is a compact subset C of G, such that Hi C CH2C and H2 C CHiC. 

1.16. Remark. Note that d is not invariant under the equivalence relation ~. For example, 
the Cartan decomposition G = KAK implies that G ^ A, but we have d{A) = dim A 7^ 
dim{AN) = d{G). 

The following two theorems state a version of the main results for even n. 

11.5'. Theorem (Oh-Witte [0W3]). Assume G = SO(2,2m), and let H be a closed, con- 
nected, subgroup of G, such that neither H nor G/H is compact. 
The homogeneous space G/H has a tessellation if and only if 

1) d{H) = 2m; and 

2) either H ^ S0{1, 2m) or H ^ Sl]{l,m). 

11.5". Theorem. Assume G — SU(2,2m), and let H be a closed, connected, subgroup of 
G, such that neither H nor G/H is compact. 

The homogeneous space G/H has a tessellation if and only if 

1) d{H) = 4m; and 
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2) either H ^SU {1,2m) or H ^Sp{l,m). 

The subgroups H that arise in Theorems 11.5' and 11.5" can also be described more 
exphcitly (cf. 11.2' and 11.2" below). 

T. Kobayashi [Kb6, 1.4] conjectured that if H is reductive and it is impossible to construct 
a tessellation of G/H by using a generalization of Lemma 9.1' (see 9.1), then G/H does not 
have a tessellation. The following lists three special cases of this general conjecture. 

1.17. Conjecture. The homogeneous spaces 

a) SO(2,2m+ l)/SU(l,m), 

b) SU(2,2m+ 1)/Sp(l,m), and 

c) SU(2, 2m + 1)/ SU(1, 2m + 1) 

do not have tessellations. 

If this conjecture is true, then, for odd n, there is no interesting example of a homogeneous 
space of S0(2,n) or SU(2,n) that has a tessellation. 

11.1'. Theorem (Oh-Witte [0W3, Thm. 1.7], lozzi-Witte). Assume 

G = S0(2, 2m + 1) or SU(2, 2m + 1), 

and let H he any closed, connected subgroup of G, such that neither H nor G/H is compact. 
If Conjecture 1.17 is true, then G/H does not have a tessellation. 

The proof of Theorem 11.1' assumes the following special case proved by R. Kulkarni [Kul, 
Cor. 2.10]. In short, Kulkarni noted that the Euler characteristic of T\G/H must both vanish 
(because the Euler characteristic of G/H vanishes) and not vanish (by the Gauss-Bonnet 
Theorem). (Other results in the same spirit, obtaining a contradiction from the study of 
characteristic classes of T\G/H, appear in [KO].) 

1.18. Theorem (Kulkarni). If n is odd, then S0(2, n)/ S0(1, n) does not have a tessella- 
tion. 

Let us give a more explicit description of the closed, connected subgroups H of SO (2, 2m) 
or SU(2,2m), such that G/H has a tessellation. This shows that if n is even, then the 
Kulkarni- Kobayashi examples (1-10) and certain deformations are essentially the only inter- 
esting homogeneous spaces of S0(2,n) or SU(2,n) that have tessellations. 

1.19. Notation [Hel, Thm. 6.5.1, pp. 270-271], [Hc2, p. 180]. Fix an Iwasawa decomposi- 
tion G = KAN. Thus, 

• is a maximal compact subgroup, 

• A is the identity component of a maximal split torus, and 

• is a maximal unipotent subgroup. 

The following two results are stated only for subgroups of AN, because the general case 
reduces to this (see 3.5). The reason is basically that H contains a connected, cocompact 
subgroup that is conjugate to a subgroup of AN. (Clearly, if H' is any cocompact subgroup 
of H, then G/H has a tessellation if and only ii G/H' has a tessellation.) This is not quite 
true in general, but the following lemma provides a satisfactory substitute, by showing that 
it becomes true after enlarging H hy a compact amount. 
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3.5'. Lemma. After replacing H by a conjugate subgroup, there is a closed, connected sub- 
group H* ofG, such that H*/H and H* /{ANnH*)° are compact, where {ANnH*)° denotes 
the identity component of AN fl H* . 

11.2'. Theorem (Oh-Witte [0W3, Thm. 1.7]). Assume G = SO(2,2m), and let H be a 
closed, connected, nontrivial, proper subgroup of AN. 

The homogeneous space G/H has a tessellation if and only if H is conjugate to a sub- 
group H' , such that either 

1) H' ^ ^0{l, 2m) r\ AN; or 

2) H' belongs to a certain family {Hb} of deformations of SU(l,m) fl AN, described 
explicitly in Theorem 9.7 [with F = M). 

11.2". Theorem. Assume G = SU(2,2m), and let H be a closed, connected, nontrivial, 
proper subgroup of AN. 

The homogeneous space G/H has a tessellation if and only if H is conjugate to a sub- 
group H' , such that either 

1) H' belongs to a certain family {-f^[c]} of deformations o/SU(l,2m) fl AN, described 

explicitly in Theorem 9.14; or 

2) H' belongs to a certain family {Hb} of deformations o/Sp(l,m) HAN, described ex- 
plicitly in Theorem 9.7 {with F = C). 

The proof of Theorem 11.2' (or 11.5') in [0W3] requires a hst [OWl] of all the homogeneous 
spaces of S0(2,n) that admit a proper action of a noncompact subgroup of SU(2,n). (The 
list was obtained by very tedious case-by-case analysis. It was extended to homogeneous 
spaces of SU(2,n) in [IW].) The following proposition (1.20) provides an a priori lower 
bound on dim if, and it turns out that the classification of the interesting subgroups of large 
dimension can be achieved fairly easily (see §10). This is the main reason that we are able 
to give reasonably short complete proofs of Theorems 11.5', 11.5", 11.1', 11.2', and 11.2". 

1.20. Proposition (see 4.12, 8.21, and 7.15). Suppose G = S0(2,n) or SU(2,n), and let 
H be a closed, connected, nontrivial subgroup of AN. If G/H has a tessellation, then 

if G — S0(2,n) and n is even] 
if G — S0(2, n) and n is odd] 
if G — SU(2, n) and n is even; 
if G — SU(2, n) and n is odd. 

1.21. Acknowledgment. This research was partially supported by a grant from the National 
Science Foundation (DMS-9801136). Much of the work was carried out during productive 
visits to the Isaac Newton Institute for Mathematical Sciences (Cambridge, U.K.); we would 
like to thank the Newton Institute for the financial support that made the visits possible. 
A.I. would like to thank the mathematics department of Oklahoma State University for 
their warm and generous hospitality and Marc Burger both for pointing out a mistake in the 
original statement and proof of Theorem 11.2 and for many enlightening conversations. 
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2. CARTAN PROJECTION AND CARTAN-DECOMPOSITION SUBGROUPS 

The main problem in this paper is to determine whether or not a homogeneous space G/ H 
has a tessellation. This requires some method to determine whether or not a given discrete 
subgroup r of G acts properly discontinuously on G/H. Y. Benoist and T. Kobayashi 
(independently) demonstrated that the Cartan projection /i is an effective tool to study this 
question. It is the foundation of almost all of our work in later sections. 

In this section, we introduce the Cartan projection, and describe some of its basic prop- 
erties. First, however, we recall the notion of a proper action (a generalization of properly 
discontinuous actions) and of a Cartan-decomposition subgroup. At the end of the sec- 
tion, we use the Cartan projection to briefly discuss the question of when there is a loss of 
generality in assuming that G is connected. 

2A. Proper actions. 

2.1. Definition ([Kb5, Defn. 2.11], [Pal, Dcfn. 1.2.2, (6)]. A topological group L of home- 
omorphisms of a topological space M acts properly on M if, for every compact subset C 
of M, 

{g e L\G r\gG is compact. 

2.2. Remark. It is important to note that a discrete group of homeomorphisms of M acts 
properly on M if and only if it acts properly discontinuously on M. 

For the special case where M = G/H is a homogeneous space, the following lemma restates 
the definition of a proper action in more group-theoretic terms. 

2.3. Lemma [Kb4, Obs. 2.1.3]. A closed subgroup L ofG acts properly on G/H if and only 
if, for every compact subset C of G, the intersection L fl {CHC) is compact. 

Proof. If C is any compact subset of G, then C = CH/H is a compact subset of G/H; 
furthermore, any compact subset of G/H is contained in one of the form C. We have 

{geL\CngC^$} = {geL\ (CH) n (gCH) ^0} 

= {geL\ge{CH){CH)-'} 

= Ln{CHC-^). 

□ 

This has the following well-known, easy consequence. 

2.4. Corollary (cf. [Kb4, Lem. 2.2(2)]). Suppose H , Hi, L, and Li are closed subgroups 
OfG. If 

• L acts properly on G/H, and 

• there is a compact subset C of G, such that Hi C CHC and Li C CLC, 
then Li acts properly on G/ Hi. 

2B. Cartan-decomposition subgroups. The following definition describes the subgroups 
to which the Calabi-Markus Phenomenon apphes (cf. Example 1.6). 

2.5. Definition. We say that is a Cartan-decomposition subgroup of G if if ~ G (see 
Notation 1.15). 
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2.6. Remark. Prom the Cartan decomposition G — KAK, we know that ^4 is a Cartan- 
decomposition subgroup. 

2.7. Remark. Any conjugate of a Cartan-decomposition subgroup is a Cartan-decomposition 
subgroup. 

2.8. Lemma (Calabi-Markus Phenomenon, cf. [Kul, pf. of Thm. A. 1.2]). IfH is a Cartan- 
decomposition subgroup of G, and T is a discrete subgroup of G that acts properly discontin- 
uously on G/H, then V is finite. 

Proof. Because if is a Cartan-decomposition subgroup, there is a compact subset C of G, 
such that CHC — G. However, from Lemma 2.3, we know that F fl (CHC) is finite. 
Therefore 

r = rnG' = rn {chc) 

is finite. □ 

The following well-known, easy fact is a direct consequence of the Calabi-Markus Phe- 
nomenon. It is an important first step toward determining which homogeneous spaces have 
tessellations. 

2.9. Corollary. If H is a Cartan-decomposition subgroup of G, such that G/H is not com- 
pact, then G/H does not have a tessellation. 

2C. The Cartan projection. 

2.10. Notation [Hel, §9.1, p. 402]. • If G is connected, let be the (closed) positive 
Weyl chamber of A in which the roots occurring in the Lie algebra of N are positive 
(cf. 1.19). Thus, ^4+ is a fundamental domain for the action of the (real) Weyl group 
of G on A. 

• In the general case, let A"*" be a closed, convex fundamental domain for the action of 
the (real) Weyl group of G on A, such that is contained in the (closed) positive 
Weyl chamber of A in which the roots occurring in the Lie algebra of N are positive. 

2.11. Definition [Hel, Thm. 9.1.1, p. 402], [Ben, Kb5]. For each element g of G, the Car- 
tan decomposition G — KA'^K implies that there is an element a of A'^ with g e KaK. 
In fact, the element a is unique, so there is a well-defined function G — > given by 

g G Kii{g)K. 

We remark that the function fi is continuous and proper (that is, the inverse image of any 
compact set is compact). 

The following crucial result of Y. Benoist provides a uniform estimate on the variation 
of fl over disks of bounded radius. (A related result was proved, independently and simul- 
taneously, by T. Kobayashi [Kb4, Thm. 3.4].) The proof is both elementary and elegant. 
However, it requires a bit of notation, so we postpone it to §8B (and, for concreteness, we 
will assume that G is either S0(2,n) or 811(2,72) in the proof). 

2.12. Proposition (Benoist [Ben, Prop. 5.1]). For any compact subset C of G, there is a 
compact subset C' of A, such that iiiCgC) C n{g)C', for all g E G. 

2.13. Notation. For subsets U and V of A'^, we write U ~ V if there is a compact subset G 
of A, such that U G VC and V G UG. 



TESSELLATIONS OF HOMOGENEOUS SPACES 11 

2.14. Corollary (Benoist [Ben, Prop. 5.1], Kobayashi [Kb4, Thm. 1.1]). For any subgroups 
Hi and H2 of G, we have Hi ~ H2 if and only if ii{Hi) /x(i?2)- 

Proof. (^) Let C be a compact subset of G, such that Hi C CH2C and H2 C CHiC. 
Choose a corresponding compact subset C of A, as in Proposition 2.12. Then 

li{Hi) c ii{CH2C) C ii{H2)C' 

and, similarly, ii{H2) C ii{Hi)C'. 

(<^=) Let C be a compact subset of A, such that /^(i^i) C ijl{H2)C and //(i?2) C ii{Hi)C. 
Then 

i/i C Kfi{Hi) K CK {ii{H2)C) K CK {{KH2K)C) K 
and, similarly H2 C KHi{KCK). □ 

The special case where H2 = G (and i/i is closed and almost connected) can be restated 
as follows. 

2.15. Corollary (Benoist, Kobayashi). H is a Cartan-decomposition subgroup of G if and 
only if ix{H) ^ ^4+. 

2.16. Corollary (cf. [Kb3, Lcm. 3.2]). Assume that M-rankG = 1. The subgroup H is a 

Cartan-decomposition subgroup of G if and only if H is noncompact. 

Proof. (4=) We have /i(e) = e, and, because fi is a proper map, we have fi{h) — ^ cxd as 
h ^ 00 in H. Because M-rankG = 1, we know that A'^ is homeomorphic to the half- line 
[0,00) (with the point e in ^4+ corresponding to the endpoint of the half-line), so, by 
continuity, it must be the case that fJ.{H) — A'^. Then Corollary 2.15 implies that if is a 
Cartan-decomposition subgroup, but we provide the following direct proof that avoids any 
appeal to Proposition 2.12. 

Prom the definition of /x, we have KHK = K fi{H) K. Therefore 

KHK = K i^{H) K = KA+K = G, 

so if is a Cartan-decomposition subgroup (by taking C = K in Definition 1.15). □ 

By using Lemma 2.3, the proof of Corollary 2.14 also establishes the following. 

2.17. Corollary (Benoist [Ben, Prop. 1.5], Kobayashi [Kb4, Cor. 3.5]). Suppose H and L 
are closed subgroups ofG. The subgroup L acts properly on GjH if and only if \i{Ij)^\i{H)G 
is compact, for every compact subset C of A. 

2D. Disconnected groups. As was mentioned in Remark 1.4, we may assume, without 
loss of generahty, that H is connected. However, it may not be possible to assume that G is 
connected, because, although there are no known examples, it is possible that the following 
question has an affirmative answer. 

2.18. Question. Does there exist a homogeneous space G/H (satisfying Assumption 1.3), 
such that G° / H° has a tessellation, but GjH does not have a tessellation? 

If r is a crystallographic group for G° /H°, then it is easy to see that r\G/H is compact. 
However, the following example shows that P may not act properly discontinuously on G/H. 

2.19. Example. Let 

• L = ii = SL(2,M), 
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• a he the automorphism of L x if that interchanges the two factors (that is, a{x, y) ~ 

• G = {L X H) X (a) (semidirect product), and 

• r be a cocompact lattice in L (cf. 1.5(2)). 

Then H = H°, and F is a crystallographic group for G°/H = {L x H)/H (see Example 1.9). 
However, T C L — a~^Ha, so F does not act properly on G/H (see 2.3 with C = {a, o^^}). 
Even so, G/H does have a tessellation, because the diagonal embedding 

A(F) = {(7,7)eLxi/|7eF} 

is a crystallographic group for G/H. Thus, this example does not provide an answer to 
Question 2.18. 

In this example, a represents an element of the Weyl group of G that does not belong to 
the Weyl group of G° . The following proposition shows that this is a crucial ingredient in 
the construction. 

2.20. Proposition. LetV he a crystallographic group for G° / H° . 

If the {real) Weyl group of G is same as the {real) Weyl group of G° , then V is a crystal- 
lographic group for G/H. 

Proof. By assumption, we may choose the same fundamental domain for the Weyl groups 
of G and G° . Let fi: G ^ A'^ and fi° : G° ^ A'^ be the Cartan projections; then fi° is the 
restriction of fi to G°. For simplicity, assume, without loss of generality, that H G G° (for 
example, assume H is connected). Then, for any compact subset G of A, we have 

At(F) n fi{H)G = /x°(F) n n°{H)G 

is finite (see 2.17). Thus, F acts properly discontinuously on G/H (see 2.17), as desired. □ 

A. Borel and J. Tits [BT, Cor. 14.6, p. 147] proved that if G is Zariski connected, then 
every element of the Weyl group of G has a representative in G°. Also, any element of the 
Weyl group must act as an automorphism of the root system. Thus, we have the following 
corollary. 

2.21. Corollary. LetT be a crystallographic group for G° / H° . 

If either 

• G is Zariski connected, or 

• every automorphism of the real root system of G° belongs to the Weyl group of the root 
system, 

then r is a crystallographic group for G/H. 

2.22. Example. 1) If G = S0(2, n), then G is Zariski connected (because SO(n + 2, C) 
is connected [GdW, Thm. 2.1.9, p. 60]), so G/H has a tessellation if and only if G°/H° 
has a tessellation. 

2) More generally, if G° = S0(2, n)° or SU(2, n) (with n > 3), then every automorphism of 
the real root system of G° belongs to the Weyl group of the root system (cf. Figure 7.1), 
so G/H has a tessellation if and only if G°/H° has a tessellation. 
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2.23. Example. If G = SL(3, M) x (a), where a is the Cartan involution of SL(3, M), then 
a represents an element of the Weyl group of G that does not belong to the Weyl group 
of G", so the proposition does not apply to G. However, this does not matter: if neither H 
nor G/H is compact, then Theorem 1.8 implies that G°/H° has no tessellations, so G/H 
has no tessellations either. 

3. Preliminaries on subgroups of AN 

This section recalls a technical result that often allows us to assume that H is a. subgroup 
of AN. It also recalls some basic topological properties of such subgroups, and also recalls 
a simple observation relating these subgroups to the root spaces of the Lie algebra q. 

3A. Reduction to subgroups of AN. 

3.1. Definition [Hel, Thm. 9.7.2, p. 431]. An element g of G is: 

• hyperbolic if g is conjugate to an element of A; 

• unipotent if g is conjugate to an element of N; 

• elliptic if g is conjugate to an element of K. 

3.2. Lemma (Real Jordan Decomposition [Hel, Lem. 9.7.1, p. 430]). Each g & G has a unique 
decomposition in the form g — auc, such that 

• a is hyperbolic, u is unipotent, and c is elliptic; and 

• a, u, and c all commute with each other. 

3.3. Remark. If g — auc is the Real Jordan Decomposition of some element g of G, then a, 
u, and c commute, not only with each other, but also with any element of G that commutes 
with g. This is because the Real Jordan Decomposition of h~^gh is 

h-^gh ^ {h-^ah){h-^uh){h-^ch) : 

if h~^gh = g, then the uniqueness of the Real Jordan Decomposition of g implies h~^ah — a, 
h~^uh = u, and h~^gh = c. 

The following observation is a generalization of the fact that a collection of commuting 
triangularizable matrices can be simultaneously triangularized. 

3.4. Lemma (cf. pf. of [Hml, Thm. 17.6]). If H is abelian {or, more generally, solvable), 
and is generated by hyperbolic and/or unipotent elements, then H is conjugate to a subgroup 
of AN. 

Because of the following result, we usually assume H C AN (by replacing H with a 
conjugate oi H'). 

3.5. Lemma (cf. [OWl, Lem. 2.9]). // H is connected, then there is a closed, connected 
subgroup H' of G and a compact, connected subgroup C of G, such that 

1) H' is conjugate to a subgroup of AN ; 

2) CH = CH' is a subgroup of G; and 

3) d{H') = d{H) {see Notation 1.12). 

Moreover, it is easy to see from {2) that the homogeneous space G/H has a tessellation if 
and only if G/H' has a tessellation. 
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Idea of proof. First, let us note that every connected subgroup of AN is closed (see 3.15(1) 
and 3.18), so we do not need to show that H' is closed. 

Second, let us note that (3) is a consequence of (1) and (2). To see this, let K* be a 
maximal compact subgroup of CH that contains C. Then a standard argument shows that 
K* n if is a maximal compact subgroup of H. (Because all maximal compact subgroups of 
CH are conjugate, there is some g e CH, such that {g~^K*g) fl if is a maximal compact 
subgroup of H that contains K* fl H. Since C C K*, we know that C normalizes K*, 
so we may assume g E H; thus, g normalizes H. Then g^^{K* fl H)g = {g^^K*g) fl H 
contains K* fl H. Because K* fl if is compact, this implies that g normalizes K* fl H. So 
K* nH — (g~^K*g) fl if is a maximal compact subgroup of if.) Therefore 

dim{K*H/K*) = dim{H/{K* n H)) = dim if - dim{K* n H) = d{H). 

Similarly dim{K*H'/K) = d{H'). Since fs:*if ^ CH ^ CH' = K*H', we conclude that 
d{H') = d{H), as desired. 

Case 1. Assume if is semisimple. We have an Iwasawa decomposition if = KhAhNh] let 
H' = AhNh and C = Kh- 

Case 2. Assume if = {/i*} is a one-parameter subgroup. Let 

• = a^u^c^ be the Real Jordan Decomposition of (see 3.2); 

• H' = {a*M*}; and 

• C = {c*} be the closure of {c*}. 

(Lemma 3.4 implies that if' is conjugate to a subgroup of AN.) 

Case 3. Assume if is abelian. We may write if as a product of one-parameter subgroups: 

H^{hl'hl^---hl^ Ih,... ,treR}. 

Let hj = OjUjC^j be the Real Jordan Decomposition of kj (see 3.2). Note that a*^, Uj, and c*^ 
commute, not only with each other, but also with every a^", u'f^, and c^" (see 3.3). Let 

H' = { {a\'u{'){aful') ■ ■ ■ (a*'-^*'-) | ti, . . . , G M }, 
and let C = {4} • • • {c\}. (Lemma 3.4 imphes that if' is conjugate to a subgroup of AN.) 

Case 4- The general case. Prom the Levi decomposition [Jac, p. 91], we know that there 
is a connected, semisimple subgroup L of if and a connected, solvable, normal subgroup it! 
of if, such that H = LR (and L fl i? is finite). Let U = [if, i?], so f/ is a connected, normal 
subgroup of if, and U is conjugate to a subgroup of N (cf. [Jac, Cor. 2.7.1, p. 51]). By 
modding out U, we (essentially) reduce to the direct product of Cases 1 and 3. □ 

3.6. Remark. For if and if' as in Lemma 3.5, Proposition 3.20 (and 3.18) imphes that if 
if' 7^ AN, then AN/H' is not compact; also. Proposition 3.15(3) (and 3.18) implies that if 
if' 7^ e, then if' is not compact. Therefore: 

• if ' = AN if and only if G/if is compact; and 

• if ' = e if and only if if is compact. 

Thus, if neither if nor C/H is compact, then if' is a nontrivial, proper subgroup of AN. 
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3B. Topology of solvable groups and their homogeneous spaces. Everything is this 

subsection is well known, though somewhat scattered in the literature. The main results 
are Propositions 3.15 and 3.20, which, together with Corollary 3.18, show that connected 
subgroups of AN and their homogeneous spaces are very well behaved topologically. Corol- 
lary 3.19, on the homology of very simple quotient spaces, is also used in later sections. 

We begin with the easy case of abehan groups. This lemma generahzes almost verbatim 
to solvable groups (see 3.15), but the proof in that generality is not as trivial. 

3.7. Lemma [Var, Thm. 3.6.2, p. 196]. Let R be a 1-connected, abelian Lie group. 

1) If H is a connected subgroup of R, then H is closed, simply connected, and isomorphic 

to M*^, for some k. 

2) // H and L are connected subgroups of R, then H H L is connected. 

3) If C is a compact subgroup of R, then C is trivial. 

Proof. Because R is abelian and 1-connected, the exponential map is a Lie group isomorphism 
from the additive group of the Lie algebra tr onto R. 

(1) Let k — dim if. Because the exponential map is a Lie group isomorphism (hence 
a diffeomorphism) , and because f) is a closed /c-submanifold of r, we know that exp([)) is a 
closed fc-submanifold of R. Of course, exp([)) is contained in i/, which is also a fc-submanifold 
of R. Because the dimensions are the same, we know that exp(()) is open in H. Also, because 
exp(f)) is closed in i?, we know that exp([)) is closed in H. Therefore 

(3.8) exp(f)) = H 

(because H is connected). Finally, we know that exp|(, is a diffeomorphism from its do- 
main 1) ~ R'^ onto its image H . 

(2) From (3.8), we have exp([)) = H and, similarly, exp(l) = L. Also, because exp is 
bijective, we have exp 1^ n exp [ = exp(f) n [). Therefore 

H (IL = exp 1) n exp [ = exp([) fl [) 

is connected. 

(3) Because is not compact (for k > 0), we know, from 3.7(1), that C° is trivial; so 
C is finite. Since R = (r, -I-) = R'^ has no elements of finite order, we conclude that C is 
trivial. □ 

As is usual in the theory of solvable groups, the main results of this section are proved 
by induction, based on modding out some normal subgroup L. To be effective, this method 
requires an understanding of the quotient space R/L. The information we need (even if L 
is not normal) comes from the following elementary observation, because i? is a principal 
L-bundle over R/L. 

3.9. Lemma. Let P be a principal H-bundle over a manifold M. 

1) If H is diffeomorphic to M", then 

(a) P is H -equivariantly diffeomorphic to M x H , so 

(b) P is homotopy equivalent to M. 

2) If M is diffeomorphic to M", then 

(a) P is H -equivariantly diffeomorphic to M x H , so 

(b) P is homotopy equivalent to H. 
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Proof. Any principal bundle with a section is trivial [Hus, Cor. 4.8.3, p. 48]. If either the 

fiber or the base is contractible, then there is no obstruction to constructing a section [Hus, 
Thm. 2.7.1(H1), p. 21], so P is trivial: P c::^ M x H. (The difi^eomorphism can be taken to be 
i7-equi variant, with respect to the natural i7-action on MxH, given by (m, h)h' = (m, hh').) 
Then the conclusions on homotopy equivalence follow from the fact that is contractible 
(that is, homotopically trivial). □ 

We recall the long exact sequence of the fibration i7 — > i? — > R/H: 

3.10. Lemma [Whi, Cor. IV. 8. 6, p. 187]. Let H he a closed subgroup of a Lie group R. 

There is a (natural) long exact sequence of homotopy groups: 

>7ri{H)^ TTiiR) ^ MR/H) ^ TToiH) ^ 7ro{R) ^ MR/H) ^ 0. 



3.11. Corollary. Let H be a closed subgroup of a 1-connected Lie group R. The homoge- 
neous space R/H is simply connected if and only if H is connected. 

Proof. Because R is 1-connected, we have 7ri(i?) = 7ro(-R) = 0, so, from (3.10), we know that 
the sequence 

0^7ri(i?///) ^7ro(//) ^0 
is exact. Thus, 7ri{R/H) = t^q{H), so the desired conclusion is immediate. □ 

As a step toward Proposition 3.15, we prove two special cases that describe the topology 
of normal subgroups. 

3.12. Lemma. If R is a 1-connected, solvable Lie group, then R is diffeomorphic to W^, for 
some d. 

Proof. We may assume the group R is nonabelian (otherwise, the desired conclusion is given 
by Lemma 3.7(1)). Then, because R is solvable, there is a nontrivial, connected, proper, 
closed, normal subgroup L of R. Since R/L is simply connected (see 3.11), and dim(it!/L) < 
dimi?, we may assume, by induction on dimR, that R/L is diffeomorphic to some R*^^. 

Therefore 

a) R is diffeomorphic to (R/L) x L and 

b) L is homotopy equivalent to R 

(see 3.9(2)). Because R is 1-connected, (b) imphes that L is 1-connected; hence, L is 

a 1-connected, solvable Lie group, so we may assume, by induction on dimi?, that L is 
diffeomorphic to some M.^^. Thus, (a) implies that R is diffeomorphic to M.^^ x M.^^ ~ 

as desired. □ 



3.13. Corollary (of proof). If R is a 1-connected, solvable Lie group, then every connected, 
closed, normal subgroup of R is 1-connected. 

The following proposition is a nearly complete generalization of Lemma 3.7 to the class of 
solvable groups. There are two exceptions: 

1) Of course, subgroups of a solvable group may not be abelian, so the conclusion in 
3.7(1) that H is isomorphic to some R'^ must be weakened to the conclusion that H is 
diffeomorphic to some M*^. 
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2) The intersection of connected subgroups is not always connected (see 3.14), so we add 
the restriction that L is normal to 3.7(2). (We remark that no such restriction is 
necessary if i? C AN , because the exponential map is a diffeomorphism from r onto R 
in this case [Dix, Sai].) 



3.14. Example. Let 
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is not connected. 



3.15. Proposition [Hc2, Thms. 12.2.2 and 12.2.3, pp. 137-138]. Let R be a 1-connected, 
solvable Lie group. 

1) If H is a connected subgroup of R, then H is closed, simply connected, and diffeomorphic 

to some W'-. 

2) // H and L are connected subgroups of R, and L is normal, then H n L is connected. 

3) If C is a compact subgroup of R, then C is trivial. 

Proof. (2) We may assume L is nontrivial, so dim(i?/L) < dimR. Thus, by induction on 
dimi?, using (1), we may assume that HL/L is a closed, simply connected subgroup of R/L. 
Then, since II/{H fl L) is homeomorphic to HL/L, we see that 

(3.16) H/ {H n L) is simply connected, 

so Lemma 3.11 implies that if fl L is connected. 

(1) Because R is solvable, there is a connected, closed, proper, normal subgroup L of R, 
such that R/L is abelian. We know that L is 1-connected (see 3.13), so, by induction on 
dim R, we may assume that every connected subgroup of L is closed and simply connected. 
Prom (2), we know that H f] L is connected, so we conclude that H (1 L is closed, and 

(3.17) 7ri{HnL)=0. 

From (3.10) (with H in the place of R, and L in the place of H), together with (3.16) 
and (3.17), we conclude that 7ri{H) = 0; that is, if is simply connected. So (3.12) implies 
H is diffeomorphic to some M^. 

Because both HL/L and H Cl L are closed, it is not difficult to see that H is closed. 

(3) Because R is solvable, there is a connected, closed, proper, normal subgroup L of R, 
such that R/L is abelian. We know that R/L is 1-connected (see 3.11), so R/L has no 
nontrivial, compact subgroups (see 3.7(3)); thus, we must have C C L. Therefore, C is 
a compact subgroup of L. Then, since L is 1-connected (see 3.13), we may conclude, by 
induction on dim R, that C is trivial. □ 
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3.18. Corollary. AN is a 1-connected, solvable Lie group. 

Proof. Because G is linear, it is a subgroup of some GL(n, M). Replacing G by a conjugate, 
we may assume that AN is contained in the group B of upper triangular matrices with 
positive diagonal entries (cf. 3.4). The matrix entries provide an obvious diffeomorphism 
from B onto x M"("-i)/2 ~ ]^nin+i)/2^ q 1-connected. Thus, Proposition 3.15(1) 

implies that AN is simply connected. □ 

The following observation will be used in Sections 4 and 9. 

3.19. Corollary. Let F he a connected subgroup of AN , and suppose we have a proper, C°° 
action of F on a manifold M. Then M and M/F have the same homology. 

Proof. Because the action is proper, we know that the stabilizer of each point of M is 
compact. However, F has no nontrivial compact subgroups (see 3.15(3)). Thus, the action 
is free. 

Because the action is free, proper, and C°°, it is easy to see that the manifold M is a 
principal fiber bundle over the quotient M/F [Pal, Thm. 1.1.3]. Furthermore, the fiber F of 
the bundle is contractible (see 3.15(1)), so Lemma 3.9(1) implies that M homotopy equivalent 
to M/F. Therefore, the spaces M and M/F have the same homology. □ 

For the special case where M/F is a homogeneous space of a solvable group, the following 
more detailed result describes the topology of M/F, not just its homology. 

3.20. Proposition (Mostow [Mos, Prop. 11.2]). // H is any connected subgroup of a 1- 
connected, solvable Lie group R, then R/H is diffeomorphic to the Euclidean space W'', 
for some d. 

Proof. Because R is solvable, it has a nontrivial, connected, closed, abehan, normal sub- 
group L. Since L is abelian and if fl L is connected (see 3.15(2)), we know that L/{H fl L) 
is a 1-connected abelian group (see 3.11), so it is isomorphic to some (sec 3.7(1)). 

We know H is closed (see 3.15(1)). Also, since L is nontrivial, we have dim(i?/L) < dimi?, 
so we may assume, by induction on dimi?, that 

R/{HL) ~ {R/L)/{HL/L) 

is diffeomorphic to some M^^. 

Now i? is a principal ifL-bundle over R/{HL). Because R/{HL) ~ R'^^^ ^j^is bundle is 
trivial (see 3.9(2)): R is iJL-equivariantly diffeomorphic to R/{HL) x HL. Then 

R/H ~ R/{HL) X HL/H ~ R/{HL) x L/{H r\ L) ^ W^"" x W^^ = ]R<^i+<^^ 

as desired. □ 

3C. T-invariant subspaces of o+n. The following well-known observation puts an impor- 
tant restriction on the subspaces of a -|- n that are normalized by a torus. It is an ingredient 
in our case-by-case analysis of all possible subgroups of AN in Sections 10 and 11. 

3.21. Lemma. Let 

• be the set of weights of A on n {in other words, the set of all positive real roots 
ofG); 

• T be a subgroup of A; 

• e $+ U {0}; 
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• n"'^ — 0o-ir=w|T ^^^'^^ ^^"^ ''^^'^ ^ U {0}' such that the restriction 
of a to T is the same as the restriction of cu to T; 

• n^"^ = 0o.|T7^tj|T ^o-; where the sum is over all a G U {O}, such that the restriction 

of a to T is not the same as the restriction of uj to T. 
If u is any M.-subspace of a + n normalized by T, then u = (u fl n""^) © (u fl n^"^). 

Proof. Since T G A, we know that the elements of Ado T are simultaneously diagonalizable 
(over M), so their restrictions to the invariant subspace u are also simultaneously diagonal- 
izable (cf. [ZS, Thms. 26 and 27 in §3.12, pp. 167-168]). Thus, u is a direct sum of weight 
spaces: 

u= 0u^. 

Vie* 

For each weight of T on u, we have 

= u n = u n n"'^, 

so 

Uj^Ij, = u n n='^ 

and 

uv, = unn^'^. 

The conclusion follows. □ 

4. Lower bound on the dimension of H 

In this section, we prove Corollary 4.12, an a priori lower bound on dim if. On the way, 
we recall a result of T. Kobayashi that will also be used several times in later sections, and 
we establish that crystallographic groups have only one end. 

4A. T. Kobayashi's Dimension Theorem. The following theorem is essentially due to 
T. Kobayashi. (Kobayashi assumed that H is reductive, but H. Oh and D. Witte [0W3, 
Thm. 3.4] pointed out that, by using Lemma 3.5, this restriction can be eliminated.) The 
proof here is based on Kobayashi's original argument and the modifications of Oh- Witte, 
but uses less sophisticated topology. Namely, instead of group cohomology and the spectral 
sequence of a covering space, we use only some basic properties of homology groups of 
manifolds (including Lemma 3.19). These comments also apply to Theorem 9.1. 

4.1. Theorem (Kobayashi, cf. [Kb2, Thm. 1.5], [Kbl, Thm. 4.7]). Let H and Hi he closed, 
connected subgroups ofG, and assume there is a crystallographis group V forG/H, such that 
r acts properly dis continuously on G/Hi. Then: 

1) We have d{Hi) < d{H). 

2) If d{Hi) > d{H), then T\G/Hi is compact, so G/Hi has a tessellation. 

Proof. By Lemma 3.5, we may assume H, Hi C AN. (So d{H) — dim H and d{Hi) — dim Hi 
(see 1.14).) 

From Lemma 3.19, we know that r\G and r\G/Hi have the same homology. Therefore 
max{ k I nkiT\G) ^ } = max{ k \ HkiVXG/Hi) ^ } < dimG/iJi, 
with equality if and only if V\G/ Hi is compact [Dol, Cor. 8.3.4, p. 260]. Similarly, we have 

max{ k \ Hk{T\G) ^ } = dimG/H. 
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Combining these two statements, we conclude (1) that dimG / H < dimG/Hi and, further- 
more, (2) that equahty holds if and only if r\G/Hi is compact. □ 

4.2. Corollary (Kobayashi). Let H and Hi be closed, connected subgroups of G, such that 
d{Hi) > d{H). If there is a compact subset C of A, such that n{Hi) C ii{H)C , then G/H 
does not have a tessellation. 

Proof. Suppose F is a crystallographic group for G/H. (This will lead to a contradiction.) 
Because F acts properly discontinuously on GjH^ the assumption on \i{H\) implies that F 
also acts properly discontinuously on GjHx (cf. 2.17). So Theorem 4.1(1) yields a contra- 
diction. □ 

4B. Crystallographic groups have only one end. It is easy to see that crystallographic 
groups are finitely generated; we now show that they have only one end (see 4.9). 

4.3. Definition (cf. [Gro, 0.2.^2, p. 4]). Let F be a finite generating set for an (infinite) 

group F. Wc say that F has only one end if, for every partition F = U U C of F into 
three disjoint sets Ai, A2., and C, such that and A2 arc infinite, but C is finite, there 
exists 7 G and f E F U F~^, such that 7/ € ^2- (This does not depend on the choice of 
the generating set F.) 

The following observation is a straightforward reformulation of Definition 4.3 (obtained 
by letting = F \ (Ai U C') and C = C" \ Ai). 

4.4. Lemma. Let F be a finite generating set for an infinite group F. // A^ and G' are 
subsets of V , such that 

• Ai is infinite, 

• G' is finite, and 

• Aif e AiU G', for every f E FU F-\ 
then the complement T \ Ai is finite. 

4.5. Remark (cf. [Coh, pp. 25-26, p. 32, and Prop. 2.14]). Definition 4.3 is often stated in 
the language of Cayley graphs: The Cayley graph of F, with respect to the generating set F, 
is the graph Cay(F; F) whose vertex set V and edge set E are given by: 

y = F; 

^ = {(7,7/) l7eF,/eFuF-^}. 

The group F has only one end if and only if, for every finite subset C of F, the graph 
Cay(F; F) \ C has only one infinite component. 

The following lemma is not difficult, but, unfortunately, we do not have a proof that is 
both short and elementary. 

4.6. Lemma (see proof of Lemma 10.11(1)). If HN = AN, then, for some x E N, the 
conjugate x'^Hx is normalized by A. 

4.7. Corollary. If d{G) — d{H) < 1, and G/H is not compact, then G/H does not have a 
tessellation. 
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Proof. It suffices to show that if is a Cartan-decomposition subgroup of G (see 2.9). 
We may assume, without loss of generahty, that H C AN (see 3.5); then 

dim// + 1 = d{H) + 1 > d{G) = dim(^A^) 

(see 1.14 and 1.13). A theorem of B. Kostant [Kos, Thm. 5.1] imphes that A^" is a Cartan- 
decomposition subgroup, so we may assume N H; then dim(i/ H N) < dimA^ — 1. 
Therefore 

dim{HN) = dim// + dim A^ - dim(// n A^) > dim// + 1 > dim(^A^). 

Hence HN — AN, so, from Lemma 4.6, we see that, after replacing // by a conjugate 
subgroup, we may assume that H is normalized by A. Then, letting a; = and T = ^4 in 
(3.21), we see that [) = (f) fl a) + (f) fl n). Since HN = AN, we have [) + n = a + n, so this 
implies that a C [); therefore H contains A. Since A is a. Cartan-decomposition subgroup 
(see 2.6), this imphes // is a Cartan-decomposition subgroup, as desired. □ 

4.8. Definition (cf. [Gro, 0.2.^42, p. 4]). A topological space M is connected at oo if every 
compact subset C is contained in a compact subset C, such that the complement M \ C is 
connected. 

4.9. Proposition. If F is a crystallographic group for G/H, then V is finitely generated 
and has only one end. 

Proof. Assume, without loss of generahty, that H C AN (see 3.5). Then H is torsion 
free, so F must act freely on G/H; therefore r\G/H is a compact manifold (rather than 
an orbifold). Because F is essentially the fundamental group of r\G/H (specifically, F = 
7Ti{T\G / H) / 7ii{G / H)) , and the fundamental group of any compact manifold is finitely gen- 
erated [Rag, Thm. 6.16, p. 95], we know that F is finitely generated. 

From the Iwasawa decomposition G = KAN, wc see that G/H is homeomorphic to 
K X [AN/ H), and Proposition 3.20 asserts that AN/ H is homeomorphic to W^, for some d. 
Obviously, we must have d = dim(ylA^) — dim//, and we may assume G/H is not compact 
(otherwise, F is finite, so the desired conclusion is obvious), so Corollary 4.7 implies that 
d > 1. Thus, we conclude that G/H is connected at oo. 

To complete the proof, wc use a standard argument (cf. [Gro, 0.2.Ci, p. 5]) to show that, 
because G/H is connected at oo and V\G/ H is compact, the group F has only one end. To 
begin, note that there is a compact subset C of G/H, such that TC = G/H. Let 

Fo = {/eF|Cn/C^0} 

(cf. [PR, (ii), p. 195]). Because F acts properly discontinuously on G/H, we know that Fq 
is finite; let F he a. finite generating set for F, such that Fq C F. 

Suppose F = Ai U ^2 U C, with \Ai\ = \A2\ ~ oo and \C\ < |oo|. (We wish to show 
there exist 7 £ Ai and f E F, such that 7/ e A2; this establishes that F has only one end.) 
Because G/H is connected at 00, there is a compact subset C of G/H, containing GC, such 
that {G/H) \ C is connected. Because CC C C , we have 

{G/H) \ C = (FC) \ C C AC U A2C. 

Because F acts properly discontinuously on G/H, we know AiC and A2C are closed (and 
neither is contained in C), so connectivity implies that AiC n A2C 7^ 0: there exist 7 e 
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and 7' e A2, such that 7C n 7'C ^ 0. Let / = 7~^7'; then 7 e Ai, 7/ = 7' e Ag, and 

cn/c = 7~'(7Cn7'C)^0, 

so / G Fo C F, as desired. □ 
4C. Walls of and a lower bound on d{H). 

4.10. Proposition. Assume M-rankG = 2. Lei 

• Li and L2 be the two walls of , and 

• T be a crystallographic group for G/H. 

If H is not compact, then there exists k G {1, 2}, such that, for every compact subset C of A, 
the intersection fi{T) fl LkC is finite. 



Proof (cf. Figure 4.1) . Suppose there is a compact subset C of A, such that each of fi{T)nLiC 
and /i(r) n L2C is infinite. (This will lead to a contradiction.) Let F be a (symmetric) finite 
generating set for F (see 4.9). We may assume C is so large that fJ^{'~fF) C fi{'j)C for every 
7 G F (see 2.12). We may also assume that C is convex and symmetric. 

Because F acts properly on G/H, there is a compact subset C of A, such that fi{H)C fl 
/i(F) C C (see 2.17). Furthermore, we may assume that ^{Li)C fl ^{L2)C C C. 

Let 

• M = U^er/^(7)C \ C, 

• Ml be the union of all the connected components of M that contain a point of Li, and 

• Ai = Tnn-\Mi). 

Then Ai is infinite (because /i(F) fl LiC is infinite). Also, for any 7 G and / G F, we 
have fii'jf) G fi{'y)C, so 7/ G U /i^^(C). Since F has only one end (see 4.9), this implies 
F \ is finite (see 4.4). Because fi{T) fl L2C is infinite, we conclude that Mi fl L2 7^ 0. 

Because /i(-ff) separates Li from L2, and every connected component of Mi contains a 
point of Li, we conclude that fj,{H)nMi ^ 0. This contradicts the fact that /i(iJ)Cn/i(F) C 
C. □ 

4.11. Corollary. Assume M-rankG = 2. Let 
1) Li anc? L2 be the two walls of A~^ ; and 
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2) Hi and H2 be closed, connected, nontrivial subgroups of G, 
such that 

for k = 1,2. If H is not compact, then any crystallographic group for G/H acts properly 
dis continuously on either G/ Hi or G/H2. 

Proof. Suppose T acts properly discontinuously on neither G/Hi nor G/H2. (This will 
lead to a contradiction.) From Proposition 2.17, we know there is a compact subset G of A, 
such that each of ix{V) fl fi{Hi)C and fi(T) H ^{H2)C is infinite. Then, since fi{Hk) ^ L^, 
we may assume (by enlarging C) that each of //(F) fl LiG and /x(F) fl L2G is infinite. This 
contradicts the conclusion of Proposition 4.10. □ 

4.12. Corollary. Assume M-rankG = 2. Let 

1) Li and L2 be the two walls of A~^ ; and 

2) Hi and H2 be closed, connected, nontrivial subgroups of G; 

such that 

n{Hk) « Lk 

for k = 1,2. If G/H has a tessellation, and H is not compact, then 

d{H) > uim{d{Hi),d{H2)}. 

Proof. The desired conclusion is obtained by combining Corollary 4.11 with Theorem 4.1(1). 

□ 

4.13. Remark. For G = SL(3,]R), there does not exist a connected subgroup H^, such that 
li{Hk) ~ Lk (cf. 6.3). Thus, Corollary 4.12 does not provide a lower bound on d{H) in this 
case. 

5. OnE-DIMENSIONAL SUBGROUPS 

Although the following conjecture does not seem to have been stated previously in the 
literature, it is perhaps implicit in [0W3]. 

5.1. Conjecture. If d{H) = 1, then G/H does not have a tessellation. 

In this section, we establish that the conjecture is valid in two cases: if either M-rankG < 2 
(see 5.14) or G is almost simple (see 5.30). Each of these illustrates a general theorem: for 
groups of real rank two, the conjecture follows from a theorem of Y. Benoist and F. Labourie 
that is based on differential geometry; H. Oh and D. Witte observed that, for simple groups, 
the conjecture follows from a theorem of G. A. Margulis that is based on unitary represen- 
tation theory. 

The following example is the only case of Conjecture 5.1 that is needed in later sections. 
(It is used in the proof of Theorem 1.8.) Because ]R-rank(SL(3, F)) = 2 and SL(3,F) is 
almost simple, this example is covered both by the theorem of Bcnoist-Labourie and by the 
theorem of Margulis, but it would be interesting to have an easy proof. 
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5.2. Proposition (see 5.14 or 5.30). Assume G = SL(3, F), for F = C, or and let 



Then G/ Hi does not have a tessellation. 
Let us begin with an easy observation. 

5.4. Lemma. If d{H) = 1 and R-rankG = 1, then G/H does not have a tessellation. 

Proof. We may assume H C AN (see 3.5). From (2.16), we know that H is a Cartan- 
decomposition subgroup, so Lemma 2.9 imphes that G/H must be compact; thus, the trivial 
group e is a crystallographic group for G/H. However, since 

d{G) = dim A + dimA^ > 1 + 1 > 1 = d{H) 

(see 1.13), and e acts properly discontinuously on G/G, this contradicts Theorem 4.1(1). □ 

5.5. Remark. The proof of Lemma 5.4 shows that the dimension of every connected, cocom- 
pact subgroup of G is at least d{G). This is a result of M. Goto and H.-C. Wang [GtW, 



5A. The geometric method of Y. Benoist and F. Labourie. 

5.6. Theorem (Benoist-Labourie [BL, Cor. 3]). If H is reductive and contains an element 
of A in its center, then G/H does not have a tessellation. 

To illustrate the idea behind Theorem 5.6, we give a direct proof of the following special 

case (under an additional technical assumption (sec 5.11)), which is sufficient for our needs. 
(Note that Condition (5.11) is satisfied for the subgroup Hi of Proposition 5.2.) Benoist and 
Labourie prove the general case by using a slightly different 1-form in place of the form uj 
that we define in Step 1. 

5.7. Corollary (Benoist-Labourie). If H is a one- dimensional subgroup of A, then G/H 
does not have a tessellation. 

Proof [BL]. Suppose F is a crystallographic group for G/H. (This will lead to a contradic- 
tion.) By passing to finite-index subgroups, we may assume that G and H are connected. 

We will construct a volume form v on T\G/H that is exact: v = d$^. The integral of u 
over r\G/H is the volume of r\G/H, which is obviously not 0, but Stokes' Theorem implies 
that the integral of any exact form over a closed manifold is 0. This is a contradiction. 

Step 1. Construction of a certain 2-form fl on T\G/H. Let m = f)^ be the orthogonal 
complement to f), under the Killing form. Then m is an (Adc -f^) -invariant subspace of q, 
such that Q — i) + m and f) n m = 0. Let : g — > f) be the projection with kernel m, and let 
u! be the corresponding left-invariant f)-valued 1-form on G. 

The space G is a homogeneous principal iJ-bundle over G/H. It is well known [KN, 
Thm. 11.11.1, p. 103] that uu is the connection form of a G-invariant connection on this 
bundle, and that the curvature form Q of this connection is given by 



(5.3) 




(1.2), p. 263]. 



(5.8) 



niX,Y) = -luj{[X,Y]) 



for X,Y em. 
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Also, because H is abelian, the Structure Equation [KN, Thm. II. 5. 2, p. 77] implies 

(5.9) Y) = duj{X, Y) for X,Y e Q. 

By identifying the 1-dimensional Lie algebra f) with M, we may think of a; and Q as ordinary 
(that is, M- valued) differential forms. 

Because u and (hence) Q are left-invariant, they determine well-defined forms ZJ and Q on 
r\G. (Note that cJ is a connection form on the principal iJ-bundlc r\G over r\G/H, and 
the curvature form of this connection is Q.) Because H is abelian, we have Qgh = for all 
h & H (cf. [KN, Prop. 11.5.1(c), p. 76]), so the horizontal form Q determines a well-defined 
form Q on the base space r\G/H. 

Step 2. Construction, for a certain s, of a certain s-form jl on T\G/H. Identifying 1) with M 
provides an ordering on the weights of i). Let 

• trio = Cg(f)) n m be the weight space of adg i) on m; 

• m+ be the sum of the positive weight spaces of adg i) on m; 

• m_ be the sum of the negative weight spaces of adg i) on m; 

• s = dim mo; and 

• /X be a nontrivial left-invariant s-form on G, such that 

(5.10) /i(Pl ©m+em_, *,...,*) = 0. 

Because is left- invariant, and /i({),*,... , *) = 0, the form determines a well-defined 
differential form fl on r\G/H. (We remark that, because g — i) (B m+ © mo © m_. Condi- 
tion (5.10) implies that the form /i is unique up to a scalar multiple.) 

Step 3. For a certain r, the wedge product pt, A Q^'' is a volume form on V\G/H. Let 
r = dimm+. It suffices to show that the restriction of Vt to m+ ® m_ is a symplectic form. 
It is obviously skew, so we need only show that it is nondegenerate. Thus, letting 

3 = { X e m+ © m_ I fi(X, m+ ® m_) = O) 

= {X e m+ © m_ I u;([X,m+ © m_]) = 0} (5.8) 

— {X E m+ © m_ I [X, m+ © m_] cm} (definition of uj) 

= { X e m+ © m_ I ({) I [X, m+ © m_])Kiiiing = } (m = f)^), 

we wish to show 3 = 0. 

There is no harm in passing to the complexification of Q. Let t*^ be a Cartan subalgebra 
of 0^ that contains f). Because t*^ preserves the Killing form, centralizes [), and normalizes 
m+ © m_, we know that 3^ is (adgC t'^)-invariant; thus, 3"^ is a sum of root spaces. 

Suppose there exists a nonzero element X of 3^, such that X belongs to some root space 0^. 
(This will lead to a contradiction.) There exists Y e g^^, such that 

(t I [X,r])Kmmg = «(t) 

for aU t ef- [Hm2, Prop. 8.3(c), p. 37]. Because 0^ C m+ © m_, we have a{^) ^ 0. Then 
(— «)(()) 7^ 0, so c m+ © m_. We now know that X e 3'^ and Y e m+ © m_, so 
({) I [X, F])Kimng = 0. We therefore conclude, from the definition of Y, that a{\}) — 0. This 
is a contradiction. 

Step 4- The form Q is exact: we may write Q = dcj). Let ujq be the connection form of a flat 
connection on r\G over T\G/ H. (Since the principal bundle is trivial (see 3.9), it is obvious 
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that there is a flat connection.) For any vector fleld X on T\G/H, let X be the hft of X to 
a vector field on r\G that is horizontal with respect to the flat connection ujq. 
Since H is abehan, there is a well-deflned 1-form on T\G/H given by 



<P{X)=uj{X). 



Then 









(deflnition of d) 


\ (^(^(^)) 


-Y{uJ{X)) 




(definition of (p) 




-Y{U{X)) 




{ujo is fiat) 


duj{X,Y) 






(definition of d) 


Tl{x,y) 






(5.9) 


n{x, Y) 






(definition of Cl). 



Assumption. For simplicity, assume that 

(5.11) every hyperbolic element of the center o/Cg(f)) is contained in i). 

Step 5. We have dji A il^(^-i) = 0. Let 

• Xi, . . . , X,. be a basis of m^, and 

• Yi, . . . ,Yrhe the dual basis of m_, with respect to the symplectic form fl on m_( 

Thus, neiXj,Yk) = dj^k- 

Let Zi,. . . , be a basis of mo, write 



[Xj ,Yk]^Yl 4,kZi (mod f) + m+ + m_ ) , 



and define 



3 3,i 



Suhstep 5.1. W is independent of the choice of the basis {Xj} o/m+ {with the understanding 
that {Yj} must be the dual basis o/m_). Let 



(5.12) 

for some a, 6 G M with a 7^ 0. Then 



aXi + bX2 if j = 1 
Xj if j > 2 



;i/a)Fi 



ifj = l 



Y;= {Y2-{b/a)Yr ifj = 2 
Yj ifi>3. 
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SO 

= ([Xi, Fi] + ib/a)[X,, Y,]) + {[X,, Y2] - ib/a)[X2, Fi]) + Yj] 

-Yl^Xj,Yj\ 

j 

= W. 

Since Xi, . . . , X,. can be transformed into any other basis of m+ by a sequence of elementary 
operations as in (5.12), we conclude that W is independent of the choice of basis, as desired. 

Substep 5.2. We have Wei). Substep 5.1 imphes that W is centrahzed by Cg{H), so W is 
in the center of Cg([)). 

Let cr be a Cartan involution of G with (T{h) = —h for /i G f). Substep 5.1 implies a{W) — 
—W. (Prom Substep 5.1, we see that W depends only on [) and the chosen identification 
of I) with R; a reverses the choice of identification.) 

Thus, W is a hyperbolic element of the center of Cg(f)). By Assumption 5.11, this implies 
W E i), as desired. 

Substep 5.3. Completion of Step 5. Let 

"V* A/"* "X/"* '7* '7* 

J^l, . . . , Ij. , Zji, . . . ,Zjg 

be the basis of (g/f))* dual to 

Xi, . . . , Xr, Yi, . . . , Yr, Zi, . . . , Zg. 

We may assume /j, = Z^ A - ■ ■ AZ*. Then, because [tUo + m_, m_] C m_ and [trio + m+, m+] C 
m.+, we have 

(iZ;(mo + m_, m_) = = dZ^{mo + m+, m+), 

so 

dZ; = -J2 (4,kX; A (mod ml Am*). 

Therefore 

{s + 1) rf/i = ^{-ly-^dZl A A • • • A A • • • A 

^Y.i-lYa]^,X*AY:AZlA---AZjA---AZ:. 

3,k,e 

Prom the choice of Yi, . . . ,Yr, we have 0,-2 YJj^i ^ so 

r 

QA(r-i) ^ 2'-i(r - 1)! ^x* A y; A • • • A x; A 17 A • • • A x; A y; 

and 

^ 2V! X* A y; A • • • A x; a y;. 
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Hence 

{s + i)dfiA i^^^'"-^) = ^ Y.i-'^y^h^i A ■ ■ ■ A z; A ■ ■ ■ A A ll^'' 

s + 2r 



iwiii A Cl 



2r 

Since W & i), we have Lwi/J'e A 0^'') = 0, so the desired conclusion follows. 
Step 6. fl A n^*" zs exact. We have 

A A Q^('^-i)) = ±d/i A Q^^''-^) A^±flAd^A n""^'-^^ ±11 A 4) A dQ^^'-^) 

= OA0±/iAQA Q^^''-^) ± A A 

= ±/i A Q^''. 

^tep 7. A contradiction. From Step 3, we know that 

/i A 0. 



'T\G/H 

On the other hand, Step 6 implies that this integral is zero. This is a contradiction. □ 

5.13. Notation [Ben, p. 320]. Let r be the opposition involution in that is, for a G A"*", 
r(a) = fi{a^^) is the unique element of that is conjugate (under an element of the Weyl 
group) to a"^ Thus, for all h E G, v/e have 

^i{h-') = r{f,{h)). 

See (6.1) for an explicit description of the opposition involution in G = SL(3, M). For 
some groups, such as G = S0(2,n), we have fi{h~^) = ix{h) for all /i G G (see 8.14); in such 
a case, the opposition involution is simply the identity map on . 

5.14. Corollary. If d{H) = 1 and M-rankG < 2, then G/H does not have a tessellation. 

Proof. Suppose F is a crystallographic group for G/H. (This will lead to a contradiction.) 

From (5.4), we know R-rankG = 2. Let Li and L2 be the two walls of A'^ and, for 
k e {1, 2}. let Hk = Lfc U L^^. Because is a ray (that is, a one-parameter semigroup), it 
is clear that is a subgroup of A. 

From Proposition 4.10, we know that there is some k G {1,2}, such that 

/i(r) n LkC is finite, 

for every compact subset G of ^4. Since F = F~^, we have t(/x(F)) = A*(r), so this implies 
that 

/x(F) n r(Lfc)G is finite, 
for every compact subset G of A. Also, because Lk C A'^, we have //(-L^) = L^, so 
fx{Hk) = fx{Lk U L-^) = ii{Lk) U T(/x(Lfc)) = Lfe U T(Lfc). 

Therefore 

//(F) n fi{Hk)C = (//(F) n LfeG)u(/i(F) n T(Lfe)G) is finite, 

for every compact subset G of A. Hence, Corollary 2.17 implies that F acts properly discon- 
tinuously on G/Hk- Then, because d{H) = 1 = d{Hk) (see 1.14), Theorem 4.1(2) implies 
that G/Hk has a tessellation. This contradicts Corollary 5.7. □ 
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5B. The representation-theoretic method of G. A. Margulis. 

5.15. Definition [Mr2, Defn. 2.2, Rmk. 2.2]. The subgroup H is tempered in G if there 
exists a (positive) function / e L^{H) (with respect to a left- invariant Haar measure on if), 
such that, for every unitary representation tt of G, either 

• |(7r(/i)0 I ■0)1 < f{h) \\4>\\\\ip\\ for all /i e if and all 7^-fixed vectors and ^0; or 

• some nonzero vector is fixed by every element of tt{G). 

For many examples of tempered subgroups of simple Lie groups, see [Oh] . 

5.16. Theorem (Margulis [Mr2, Thm. 3.1]). If H is noncompact and tempered, then G/H 
does not have a tessellation. 

Proof [Mr2] . Suppose F is a crystallographic group for G/H. (This will lead to a contradic- 
tion.) To simplify the notation somewhat (and because this is the only case we need), let us 
assume H = {h^} is a one-parameter subgroup of G. 

Because G and F are unimodular (recall that G is semisimple (see 1.3) and F is discrete), 
there is a G-invariant measure (in fact, a G-invariant volume form) on the homogeneous 
space F\G [Rag, Lem 1.4, p. 18]. Thus, the natural representation tt of G on L^(F\G), 
defined by 

(7r(^)0)(x) = 0(x^-i), for e L2(F\G), geG^xE F\G, 

is unitary. 

Because H is noncompact, and acts properly on F\G, we know that any compact subset 
of F\G has infinitely many pairwise-disjoint translates (all of the same measure), so we see 
that 

(5.17) F\G is not compact, and has infinite volume. 

Therefore, tt has no nonzero G-invariant vectors, so, because H is tempered, we know that 
there is some / e i^^(IR), such that 



(5.18) m\m2m2> 



(f){xh*) ip{x) dx 



for alH G M and all JC-invariant 0, -0 G L^(F\G). 

Because T\G/H is compact, there is a compact subset C of G, such that TCH = G; let 
G be the image of G in F\G. Prom the choice of G, we know, for each x G F\G, that there 
is some G R, such that 

(5.19) xh'^^ eC. 

Because Ut=o Ch^K is a compact subset of F\G, there is a positive, continuous function <j) 
on F\G with compact support, such that 

(5.20) (j){xh^) > 1 for all a; G G and all t G [0, 1], 

and, by averaging over K, wc may assume that (f) is X-invariant. 

Fix some large T G Because 1J^=^(t+i) Ch^K is compact, and F\G has infinite volume 
(see 5.17), there is some X-invariant continuous function 0^ on F\G, such that ||V't||2 = 1, 

(5.21) < ^Pt{.x) < 1 for all x G F\G, 
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and 

(5.22) |T^| > T + 1 for all X in the support of i/jt- 
We have 

/ f{t)dt> / (j){xh*) iprix) dt dx ((5.18) and HV': 

J\t\>T Jr\GJ\t\>T 

• 1 



T 2 



1) 



> / / (p{xh'^^')'il]T{,x)dtdx (5.22) 

JV\G Jo 

>[ ^T{x)dx ((5.19) and (5.20)) 
Jr\G 

> 1 ((5.21) and ||V't||2 = 1). 

However, because / e -^^(M), we know that liniT-^oo J\t\>T /(^) ~ ^- ^^^^ is a contradiction. 

□ 

We state the following well-known result of representation theory without proof. As is 
explained in [KS, §3, p. 140], it can be obtained by combining work of R. Howe [How, 
Cor. 7.2 and §7] and M. Cowling [Cow, Thm. 2.4.2]. (The assumption that M-rankG > 2 
can be relaxed: it suffices to assume that G is not locally isomorphic to S0(1, n) or SU(1, n).) 

Fix any matrix norm ||-|| on G; for example, we may let \\g\\ — maxj^fe \gj,k\- 

5.23. Theorem (Cowling, Howe). IfG is almost simple, anc? R-rankG > 2, then there are 

constants C > and p > such that, for every unitary representation tt of G, either 

1) I {n{g)(f) \ ip) \ < C\\(f)\\ \\tp\\ \\g\\^^ for all g E G and all n{K)-fixed vectors (p and ip; or 

2) some nonzero vector is fixed by every element of t:{G). 

Although we cannot prove Theorem 5.23 here, we present an elementary proof of the 
following related result, which, unfortunately, is qualitative, rather than quantitative. On 
the other hand, this simple result applies to all vectors, not only the i^-fixed vectors, and 
it applies to all semisimple groups, including SO(l,n) and SU(l,?i). It was first proved by 
R. Howe and C. Moore [HM, Thm. 5.1] and (independently) R. Zimmer [Zml, Thm. 5.2]. 

5.24. Theorem [Zm2, Thm. 2.2.20, p. 23]. // 

• G is connected and almost simple; 

• 71 is a unitary representation of G on a Hilbert space Ti, such that no nonzero vector 
is fixed by tt{G); and 

• {dj} ^ sequence of elements of G, such that \\gj\\ — > oo, 
then {Tr{gj)(j) | ■0) — > 0, for every (pjip & Ti. 

Proo/ (Ellis- Nerurkar [EN]). Case 1. Assume {gj} C A. By passing to a subsequence, we 
may assume '^{gj) converges weakly, to some operator that is, 

{TT{gj)4> I ip) — >■ I ip) for every (f),tp eH. 

Let 

(5.25) U = {v eG \ gj^vgj ^ e } and U~ = {u e G \ gjugj^ ^ e }. 

For u G , we have 

{ET:{u)(t) I ip) = lim(7r(^jM)0 | -0) = lim{7r{gjug]'^)7r{gj)(l) \ ip) = lim(7r(^j)(/) | ip) = {Ect) \ 
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SO Et^{u) = E. Therefore E{{n^~)^) = 0. 
We have 

{E*(t> I V') = (0 I Ei;) = hm(0 | 7r(^,-)V') = \M<9j^)<t> I V'), 
so the same argument, with E* in the place of E and gj^ in the place of Qj, shows that 

E*{{n^)'-) = 0. 

Because tt is unitary, we know that 7r(gij) is normal (that is, commutes with its adjoint) 
for every j; thus, the limit E is also normal: we have E*E = EE* . Therefore 

\\E(bf = {Ecp I E(p) = {{E*E)(t) I (p) = {{EE*)(j) \ 0) = | E*(j)) = 

so ker E = ker E* . 
Thus, 

ker£; = ker£; + kerS* D {H^')^ + {H^')^ = {H''' n V}")^ = {H'^^'^'^)^. 

By passing to a subsequence of {gj}, we may assume {U, U~) = G (see 5.27). Then H^^''^ ^ = 
= 0, so ker E dQ-^ ^H. Hence, for all 4>,i) eH, we have 

lim(7r(<7j)0 I V-) = (^0 I V-) = (0 I ^) = 0, 

as desired. 

Case 2. The general case. From the Cartan Decomposition G = KAK, we may write 
gj = Cjajc'j, with Cj, c'j e K and aj e A. Because K is compact, we may assume, by passing 
to a subsequence, that {cj} and {c^} converge: say, Cj — > c and c^- — > c'. Then 

lim(7r(5'j)0 | ■0) = \im{Tr{cjajc'j)(l) \ ip) 

— lim(7r(aj)7r(c^)0 | n{cj)~'^ip) 

= lim(7r(aj)(7r(c')0) | 7r(c)"V> 
= 0, 

by Case 1. □ 

The following example illustrates Lemma 5.27. 

5.26. Example. Let G = SL(3,M), define Hi as in Proposition 5.2, and suppose {gj} is 
some sequence of elements of Hi, such that \\gj\\ co. We may write 

/e*i 
9j= \ e*^ 
\ 

where tj e IR. By passing to a subsequence, we may assume that either tj — > oo or tj — > — oo. 
If tj — > oo, then, in the notation of (5.25), we have 



[/ = < 1 * > and U' 





if tj —oo, then U and U~ are interchanged. Thus, in either case, u is the sum of two root 
spaces of q, and u~ is the sum of the two opposite root spaces. It is not difficult to see that 
[u, u~] is the sum of o and the remaining two root spaces. Therefore, we have (u, u~) = g, 
so {U, U-) = G. 
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5.27. Lemma. If G and {gj} are as in Theorem 5.24, o.'<^d {gj} C A, then, after replacing 
{gj} by a subsequence, we have {U, U~) — G, where U and U~ are defined in {5.25). 

Proof. By passing to a subsequence, we may assume {gj} is contained in a single Weyl 
chamber, which we may take to be . Then, by passing to a subsequence yet again, wc 
may assume, for every positive real root a, that either cx{gj) — > oo or cx{gj) is bounded. Let 

• be the set of positive real roots; 

• A be the set of positive simple real roots; 

• = { a e I ot{gj) is bounded }; and 

• T = n^e^ ker = ny,g*nA ker ■0. 

There is a compact subset C of A, such that {gj} C CT, so, because \\gj^ — > oo, we know 
that T is not trivial. 

For each real root a, let Uq be the corresponding root subspacc of Q. Then 

Now, for a G we have a G ^' if and only if a is in the linear span of fl A. Thus, we see 
that u is precisely the unipotent radical of the standard parabolic subalgebra p = C0(T) +o+n 
corresponding to the set ^ fl A of simple roots [BT, 4.2, pp. 85-86]. 

Similarly, u~ is the unipotent radical of the opposite parabolic algebra p~ = Cg(T) + o+n~. 
Because G is simple, the unipotent radicals of opposite parabolics generate q [BT, Prop. 4.11, 
p. 89], so ([/, U ) = G, as desired. □ 

5.28. Corollary (of Theorem 5.23). Assume G is simple, and M-rankG > 2. If H is a 

one-parameter subgroup of AN, then either 

1) H is tempered; or 

2) H gN. 

Proof [OW 3, Prop. 3.7]. Write H = {h*}. From the Real Jordan Decomposition (3.2), we 
may assume, after replacing if by a conjugate subgroup, that /i* = a*w*, where a* G ^ is 
a hyperbolic one-parameter subgroup, and & N is a unipotent one-parameter subgroup, 
such that a* and commute with each other. 

We may assume H (/i N, so a* is nontrivial. Since the growth of the hyperbolic one- 
parameter subgroup a* is exponential, while that of the unipotent one-parameter subgroup 
is polynomial, there is some e > 0, such that 

\\h^\ = ||aV|| > viM > e''*' 

for large t G R. Since the function C/e^'l*! is in L^{R), it follows from Theorem 5.23 that H 
is tempered, as desired. □ 

5.29. Lemma. If d{H) = 1 and H (Z N, then G/H does not have a tessellation. 

Proof [0W3, Prop. 3.7]. We have dimiJ = d{II) = 1 (see 1.14), so if is a connected, one- 
dimensional, unipotent subgroup. Hence, the Jacobson-Morosov Lemma [Hel, Thm. 9.7.4, 
p. 432] implies that there exists a connected, closed subgroup iii of G, such that Hi con- 
tains if, and ifi is locally isomorphic to SL(2,]R). Then if is a Cartan-decomposition 
subgroup of ifi (see 2.16), so there is a compact subset C of A, such that /u(ifi) C fi{II)C 
(see 2.12). Also, we have d{Hi) = 2 > 1 = d{H). Therefore, Theorem 4.2 applies. □ 
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5.30. Corollary (Oh-Witte [0W3, Prop. 3.7]). If d{H) ^ 1 and G is simple, then G/H 
does not have a tessellation. 

Proof. We may assume H C AN (see 3.5), so dimif = d{H) — 1 (see 1.14). 

• If R-rankG < 2, then Lemma 5.4 applies. 

• If H G N, then Lemma 5.29 applies. 

• If M-rankG > 2 and H N, then Corollary 5.28 implies that H is tempered, so 
Theorem 5.16 applies. 

□ 



6. HOMOGENEOUS SPACES OF SL(3,M), SL(3,C), AND SL(3,H) 

Y. Benoist [Ben, Cor. 1] and G.A. Margulis (unpublished) proved (independently) that 
SL(3, M)/ SL(2, M) does not have a tessellation. Using Benoist's method, H. Oh and D. Wittc 
[0W3, Prop. 1.10] generalized this result by replacing SL(2,M) with any closed, connected 
subgroup H, such that neither H nor SL(3, M)/if is compact. The same argument applies 
even if IR replaced with either C or H. However, the proof of Benoist (which apphes in a 
more general context) relics on a somewhat lengthy argument to establish one particular 
lemma. Here, we adapt Benoist's method to obtain a short proof of Theorem 1.8 that avoids 
any appeal to the lemma. 

6.1. Notation. Assume G = SL(3,F), for F = M, C, or H. 

• Let r be the opposition involution in ^4+ (see 5.13); 

• Let B+ = {ae A+ \ r(a) = a }. 

More concretely, we have 



A+ 




ai, a2, as G 

Ol > 02 ^ O3 




O^^ 






a > 1 



6.2. Lemma. If G — SL(3, F) and d{H) — 1, then G/H does not have a tessellation. 

Proof. Since M-rankG = 2, the desired conclusion follows from Corollary 5.14; since G is 
simple, it also follows from Corollary 5.30. However, we give a proof that requires only the 
special case described in Proposition 5.2, rather than the full strength of (5.14) or (5.30). 

Suppose r is a crystallographic group for G/ H . (This will lead to a contradiction.) Let Li 
and L2 be the two walls of From Proposition 4.10, we know that there exists k G {1, 2}, 
such that /i(r)C fl is finite, for every compact subset G of A. 

Because P"^ = P, we have r(//(P)) = A*(P)- On the other hand, r interchanges Li and L2. 
Thus, the preceding paragraph implies that //(P)C fl (Li U L^) is finite, for every compact 
subset C of ^4. 
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Figure 6.1. Construction of t). 

iimu ))=xiiimo,t ))) 




Figure 6.2. Proposition 6.3: fi{H) is on both sides of B^, so it must contain B^. 

For Hi as in (5.3), we have fi{Hi) = Li UL2, so the conclusion of the preceding paragraph 
implies that F acts properly discontinuously on G/ Hi (see 2.17). Now Theorem 4.1(2) implies 
T\G/Hi is compact; thus, G/Hi has a tessellation. This contradicts Proposition 5.2. □ 

For completeness, we include the proof of the following simple proposition. 

6.3. Proposition [0W3, Prop. 7.3]. Assume G = SL(3,F). If H is a closed, connected 
subgroup of AN with dim if > 2, then B^ C ^{H). 

Proof (cf. Figure 6.2 and proof of 8.19). Since H C AN and dimif > 2, it is easy to con- 
struct a continuous, proper map [0,1] x — >■ H such that $(l,t) = $(0,t)~^, for all 
t G M"^ (cf. Figure 6.1). For example, choose two linearly independent elements u and v of (), 
and define 

$(s, t) = exp (t cos{7is)u + t sin(7rs)t') . 

If we identify A with its Lie algebra o, then A"*" is a convex cone in a and the opposition 
involution r is the reflection in ^4+ across the ray B^ . Thus, for any a G the points a 
and r(a) are on opposite sides of 5+, so any continuous curve in ^4+ from a to r(a) must 
intersect B^ . In particular, for each t G M^, the curve 

{^l{^{s,t)) |o<s<i} 

from /i($(0, t)) to /^(^(l, t)) must intersect B^ . Thus, we see, from an elementary continuity 
argument, that /i[$([0, 1] x M^)] contains B^ . Therefore, B^ is contained in ij,{H). □ 
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Figure 6.3. Proof of Theorem 1.8: fi{T) stays away from B^, because C 
f^{H). Also, half of fi(T) is on each side of B~^, because r(/i(r)) = /i(r). This 
contradicts the fact that F has only one end. 

Proof of Theorem 1.8 (cf. Figure 6.3 and proof of 4.12). Suppose F is a crystallographic 
group for G/H. (This will lead to a contradiction.) We may assume H C AN (see 3.5). 

Let F be a (symmetric) finite generating set for F, and choose a compact, convex, sym- 
metric subset C of A so large that 

fi{jF) C /i(7)C 

for every 7 G F (see 2.12). 

From Lemma 6.2, we know that dim if > 2, so Proposition 6.3 implies that B~^ C ^{H). 
Then, because F acts properly on G/H, we conclude that /i(F) H B^C is finite (see 2.17). 
Since /i is a proper map, this implies that F fl ^~^{B^C) is finite. 

Let Ai and A2 be the two components of A'^ \ B^. Because F~^ = F, we know that 
r(/i(F)) = /i(F). Then, because r interchanges Ai and A2, we conclude that r(/i(F)ny4i) = 
fj,(T) n A2. Therefore, /i(F) H Ai and /u(F) H A2 have the same cardinality, so they must both 
be infinite. So 

each of F n fi^^{Ai) and F fl fi^^{A2) is infinite. 
Because F has only one end (see 4.9), this implies there exist 

7G {Tn^i'\A,))^^i-\B+c), 

such that 

(6.4) 7/G (rn/x-^(A2)) \/i-^(i?+C), 

for some f E F. Then fi{j) G Ai, /i(7/) G A2, and 

/x(7/) e fii^F) C /x(7)C. 

Using the fact that G is symmetric and the fact that G contains the identity element e, we 
conclude that 

/^(7) e {fi{lf)G) n and /i(7/) G {Kif)c) n A2; 

therefore fi{'jf)G intersects both Ai and Since B~^ separates Ai from A2, and C is 
connected, this implies that ii{^f)G intersects 5+; hence f^i'jf) G i?"*'C. This contradicts 
the fact that 7/ ^ /i-i(5+C) (see 6.4). □ 
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7. Explicit coordinates on so(2,n) and su(2,n) 

From this point on, we focus almost entirely on S0(2, n) and SU(2, n). (The only exception 
is that some of the examples constructed in Section 9 are for other groups.) In this section, 
wc define the group SU(2,n;F), which allows us to provide a fairly unified treatment of 
S0(2,n) and SU(2,n) in later sections. 

7A. The group SU(2,n;F). 

7.1. Notation. 

• We use F to denote either M or C. 

• Let q — dim^F, so q e {1, 2}. 

• We use Fimag to denote the purely imaginary elements of F, so 



F- = 

^ imag 



if F = M 
iR ifF = C. 



For G F, there exist unique Re0 G M and Im0 G Fimagj such that = Re0 + Im0. 
(Warning: in our notation, the imaginary part oi a + bi is bi, not b.) 
For G F, we use to denote the conjugate Re — Im of 4>. (If F = M, then = 0.) 
For a row vector x G F"^~^, or, more generally, for any matrix x with entries in F, we 
use x^ to denote the conjugate-transpose of x. 



7.2. Notation. For 



/O 







1 

VI 



Id 



1\ 

1 







0/ 



G SL(n + 2,F), 



we define 

and 

Then: 



SU(2,n;F) = {g e SL(n + 2,F) | gjg^ = J} 
5u(2, n; F) = { M G sl{n + 2,¥) \ uJ + Ju^ = }. 



• SU(2,n;M) is a realization of S0(2,n), 

• SU(2, n; C) is a realization of SU(2, n), and 

• 5u(2, n; F) is the Lie algebra of SU(2, n; F). 

We choose 

• A to consist of the diagonal matrices in SU(2, n; F) that have nonnegative real entries, 

• N to consist of the upper-triangular matrices in SU(2, n; F) with only I's on the diag- 
onal, and 

• = SU(2,n;F)nSU(n + 2). 
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A straightforward matrix calculation shows that the Lie algebra of AN is 



(7.3) 



a + n 



' f ti (f) X 

h y 







y 

-t2 





X \ 



-X' 



-t 



1/ 



0,r^ e F, 
x,y e F"-2, 

X, y e Fimag 



> . 



7.4. Remark. From (7.3), we see that the first two rows of any element of a + n are sufficient 
to determine the entire matrix. In fact, it is also not necessary to specify the last entry of 
the second row of the matrix. 

7.5. Remark. Prom (1.13) and (7.3), we see that d(SU(2, n; ¥)) = dim(a + n) = 2qn. 

7.6. Notation. Because N is simply connected and nilpotent, the exponential map is a 
diffeomorphism from n to N (indeed, its inverse, the logarithm map, is a polynomial [Hcl, 
Thm. 8.1.1, p. 107], so each clement of N has a unique representation in the form expu with 
u E n. Thus, each element h of N determines corresponding values of 4>, x, y, rj, x and y 
(with ti = ^2 = 0). We write 

(t>h,Xh,yh,rih,^h,yh 

for these values. 

7.7. Notation. We let a and P be the simple real roots of SU(2,n;F), defined by 

a{a) = ai,i/a2,2 and /9(a) = 0,2,2? 
for a (diagonal) element a of A. Thus, the positive real roots (see Figure 7.1) are 

' a,P,a + P,a + 2p, if F 

^a,P,a + P,a + 2p,2p,2a + 2p if F = C. 

Concretely: 

• the root space Uq, is the 0-subspace in n, 

• the root space ng is the y-subspace in n, 

• the root space Ua+p is the x-subspace in n, 

• the root space 11^+2^ is the ?7-subspace in n, 

• the root space 1X2/3 is the y-subspacc in n (this is if F = M), and 

• the root space 1x20+2/3 is the x-subspace in n (this is if F = K) . 



7.8. Definition. Let 



tta+2^ + n2a+2/3 + ^2/3 

{zen\(j)^^0,x^^y^ 

^0 77 X 
y -77 



= 0} 

rj e¥, 
X, y G Fimag 



and, for a given Lie algebra f) C n. 



38 



ALESSANDRA lOZZI AND DAVE WITTE 




(a) 



(b) 



Figure 7.1. The real root systems of (a) S\J{2,n;'. 
(b) SU(2,n;C) = SU(2,n). 



S0(2,n) and 



Note that if 
(cf. 7.22). 



for every m G f), then C Dfj and Of, is contained in the center of i) 



7.9. Remark. By definition (see 2.10), we have 

A+ = {aeA \ a{a) > 1, (3{a) > 1 }. 
Therefore, from the definition of a and (3 (see 7.7), we see that 
(7.10) A+ = {ae A\ai^i>a2,2>l}- 

7.11. Remark. For F = H, the division algebra of real quaternions, the group SU(2,?2;EI) is 
a realization of Sp(2,n). Most of the work in this paper carries over, but the upper bound 
on dim if given in Theorem 10.19 is not sharp in this case (and it does not seem to be easy 
to improve this result to obtain a sharp bound). Thus, we have not obtained any interesting 
conclusions about the nonexistence of tessellations of homogeneous spaces of Sp(2,n). 

7B. The subgroups SU(1, n; F) and Sp(l, m; F). We now describe how the four important 
families of homogeneous spaces of Example 1.10 are realized in terms of SU(2, n; F). 

7.12. Definition. Let 

• SU(l,n;M) = SO(l,n); 

• Sp(l,n;R) = SU(l,n); 

• SU(l,n;C) = SU(l,n); and 

• Sp(l, n; C) = Sp(l, n). 

Then, for an appropriate choice of the embeddings in Example 1.10, we have 



(7.13) 



su(l,n;F) n (a + n) 



' t (f) X (j) 






imag 
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and (if 2m < n) we have 
(7.14) sp(l,m;F) n (o + n) = 

't Xi X2 Xs X4 

t —X2 xi —xl xi 



X2m-3 X2m-2 
—X2m-2 X2m-3 



77 X 

—X —fj 



t G R, 

Xj e ¥, 

X G F, 



imag 



7.15. Remark. Prom (1.13), (7.13) and (7.14), we see that 

• (i(SU(l, n; F)) = dim(su(l, n; F) n (a + n)) = qn and 

• d(Sp(l, m; F)) = dim(sp(l, m; F) n (a + n)) = 2qm. 

7C. Formulas for exponentials and brackets. The arguments in later sections often 
require the calculation of expw, for some w e n, or of [u,v], for some u,v & n. We now 
provide these calculations for the reader's convenience. 

7.16. Remark. For 



u 



/o 





X 


V 


X \ 








y 


y 


-V 











-y' 
















-0 


^0 











/ 



e n. 



we have 
(7.17) 



exp{u) 








x+ \(p% 


t rj - \xy^ + \<py - \<p 


l|^|2 

y? 

+ (x- 


-Re(0^) + ^|0P|yP\ 
-i(/)y0 + |lm(xyt0)) 





1 


y 


y-\\y? 


-77- 


||/a;t-iy0+|||/|20 








Id 


-y' 




—x^ + 











1 






Vo 













1 / 



When = 0, this simplifies to: 



(7.18) 



exp(M) 



1 



X 77 






1 


y 


y-ilyP 








Id 


—y^ —x^ 











1 


^0 








1 
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Similarly, when y = 0, we have 



(7.19) 



exp{u) 



7.20. Remark. For 



'0 



(7.21) 
we have 
(7.22) 



u 



/ 

\ 





X 


77 + 


1 1^,12 

2 l-^l 

+ (x- 


- tie[(pr]) \ 
1 ^. 





1 





y 




-w 








Id 


















1 






\o 













1 / 



(p X 1] 

y y 



and 



u 



(p X 7] 

y y 



[u, u] = 



''0 (f)y — (py —xy^ + xy^ + ipy — ipy — 2 lm{xx^ + (pfj — (prjy 



-2Im(yyt) 



yx^ — yx^ + 90 — y0 



7.23. Remark [Var, Eq. (2.13.8), p. 104]. For u,v e[), we have 

exp{-v)uexp{v) =u+[u,v] + ^[[u,v],v] + [[u,v],v],v +■■■. 

Combining this with (7.22) allows us to calculate the effect of conjugating by an element 
of A^. 

For example, suppose m G n, with 0„ = and yu = 0, and suppose v e xXa+p- We see, 
from (7.22), that = and that X[u^y] = y[u,v] = 0, so [[m, u],!;] = (see 7.22). Therefore 

exp{—v)u exp{v) = u + [u,v]. 

8. Calculating the Cartan projection 

Y. Benoist [Ben, Lem. 2.4] showed that calculating values of the Cartan projection is no 
more difficult than calculating the norm of a matrix (see 8.11). In this section, we describe 
this elegant method and some of its consequences, in the special case G = SU(2,n;F). 

8.1. Standing assumptions. Throughout this section, we assume G = SU(2,n;F). 
8A. The basic definitions. 

8.2. Notation. Wc employ the usual Big Oh and little oh notation: for functions /i, /2 on 
a subset X of G, we say 

fi = 0{h) forheX 

if there is aconstant C, such that, for ellh E X with \\h\\ large, we have ||/i(/i)|| < G\\f2{h)\\. 
(The values of each fj are assumed to belong to some finite-dimensional normed vector space, 
typically either C or a space of complex matrices. Which particular norm is used does not 
matter, because all norms are equivalent up to a bounded factor.) We say 

/i - o(/2) for hex 
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if 11/1(^)11/11/2(^)11 — > as /i — > 00. (We use /i — > oo to mean \\h\\ 00.) Also, we write 

/1X/2 

if/i = 0(/2) and/2 = 0(/i). 

We use the following norm on SU(2,n;F), because it is easy to calculate. The reader is 
free to make a different choice, at the expense of changing = to x in a few of the calculations. 

8.3. Definition. For h G SU(2,n;F), we define \\h\\ to be the maximum absolute value 
among the matrix entries of h. That is, 

l<j,k<n+2 

8.4. Definition. Define p: SU(2,n;F) ^ GL(F"+2 a F"+2) by p{h) = h A h, so p is the 
second exterior power of the standard representation of SU(2,n;F). Thus, we may define 
\\p{h) II to be the maximum absolute value among the determinants of all the 2x2 submatrices 
of the matrix h. That is. 



\\p{h)\\ = max 

l<j,k,e,m<n+2 



From (7.17), (7.18), and (7.19), it is clear that the 2x2 minor in the top right corner is 
often larger than the other 2x2 minors, so we give it a special name. 

8.5. Definition. For h e Mat„+2(F), define 

A(/i) = detf^'"+^ ^i,n+2y 

\lT'2,n+l lT'2,n+2 J 

SB. Y. Benoist's method for using matrix norms to calculate /i. 

8.6. Lemma. For a e A'^ , we have \\a\\ = ai^i and ||p(a)|| = 01,102,2- 
Proof. From (7.3), we see that 

{1 if 3 < j < n 

1/^2,2 ifj = n + l 
l/ai,i ifj = n + 2. 

Thus, from (7.10), we see that 

ai,i > a2,2 > %,j 

for J > 3 (and, since a is diagonal, we have aj^k = for j ^ k). Therefore, the desired 
conclusions follow from the definitions of ||a|| and ||p(a)||. □ 

8.8. Proposition (Benoist, cf. [Ben, Lem. 2.4]). We have 

(8.9) p{h) X h, 

(8.10) p{p{h))^p{h), 

(8.11) p{h),,, X \\hl and p{h)2,2 X \\pih)\\/\\hl 
for he SU(2,n;F). 
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Proof. Choose /ci,/c2 £ K, such that ^{h) — kihk2. Because \\xy\\ = 0(||a;|| |||/||) for x,y e 
SU(2,n;F), and maxfeg;<:||A;|| < oo (since K is compact), we have 

Mh)\\^\\hhk,\\^o{\\h\\) 

and 

11/^11 = ||^rV(/^)A;,-^ll=0(M/^)||), 
so (8.9) holds. Similarly, we have 

||p(M/^))|| = ||p(W%(MI|x|IpWII, 

so (8.10) holds. 

For a e we know, from (8.6), that Oi^i = ||a|| and 02,2 = ||p(o)||/ai,i. Thus, letting 
a — n{h), and using (8.9) and (8.10), we see that 

i,{h),,,^Mh)\\^\\h\\ 

and 

\p{m)\\ \\pm 



2,2 



as desired. □ 

8.12. Remark. Proposition 8.8 generahzes to any reductive group G [Ben, Lem. 2.3]. How- 
ever, one may need to use a different representation in the place of p. In fact, if M-rank G = r, 
then r representations of G are needed; for G = SU(2, n; ¥), we have M-rankG = 2, and the 
two representations we use are p and the identity representation I{h) = h. 

8.13. Corollary. Let — > 00 and /i„ — > 00 &e two sequences of elements 0/ SU(2, n; F). 
We have 

Qn X hn and p{gn) X p{hn) 

if and only if 

there is a compact subset C of A, such that, for all n e Z"*", we have ii{gn) G p{hn)C. 

Proof. (^) Let a = ii{hn)~^ lJ>{gn) ■ From (8.11), we see that lJ>{gn)j,j x lJ.{hn)j,j for j e {1, 2}, 
so, using (8.7), we have 

{Oil) ifl<i<2; 
1/1 = 1 if 3 < j < n; 

P'{hn)2,2/ P{9n)2,2= 0{l) if J = 71 + 1; 

^ Pihn) 1,1 /p{9n) 1,1 = 0(1) if j = n + 2. 

Therefore a = 0(1), as desired. 

{<=) Because C is compact, we have 

K9n) X ii{hn) and p{n{gn)) X p{ii{hn)) 
(cf. proof of (8.9) and (8.10)). Then the desired conclusions follow from (8.9) and (8.10). □ 

Proof of Proposition 2.12 for G — SU(2,n;F). Because G is compact, we have g' >^ g and 
p(g') X p(g) for any g' G GgG (cf. proof of (8.9) and (8.10)). Thus, the desired conclusion 
follows from Corollary 8.13. □ 



^ _ P(9n)j,j 
3,3 
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Figure 8.1. Proposition 8.19: if /^(-ff) contains a curve near each wall of A"*", 
then it also contains the interior. 

Because of Proposition 8.8, we will often need to calculate \\h\\ and The following 

observation and its corollary sometimes simplifies the work, by allowing us to replace h 
with h~^. 

8.14. Lemma. We have fi{h^^) = fi{h) for h e SU(2,n;F). 

Proof. Define J as in (7.2), and choose ki, ^2 G K, such that fi{h) = kihk2. For any a G A"*", 
we see, using (7.3) or (8.7), that Ja~^J = a, so 

{Jk^^)h'\k^^J) = Jfi{h)-^ J = fi{h). 

Note that det J = 1. Also, we have = Id and J'^ = J, so it is obvious that JJJ^ = J and 
JJ^ = Id. Therefore 

J G SU(2, n; F) n SV{n + 2) = K. 
Thus, from the definition of fi, we conclude that fi{h~^) = fi{h), as desired. □ 

The following corollary is obtained by combining Lemma 8.14 with Corollary 8.13. 

8.15. Corollary. We have h-^ x h and p{h-^) x p{h) for h G SU(2,n; F). 
8C. The walls of A+. 

8.16. Notation. For k G {1,2}, set 

(8.17) Lfc = {aG |a2,2 = <l'}. 

From (7.10), we see that Li and L2 are the two walls of A^. From (8.6), we have 

(8.18) p(a) X ||af for a G Lfc. 

We reproduce the proof of the following result, because it is both short and instructive. 
(Although we have no need for it here, let us point out that the converse of this proposition 
also holds, and that there is no need to assume H C AN.) Because of this proposition (and 
Corollary 2.9), Section 10 will study the existence of curves such that x for 
A;g{1,2}. 

8.19. Proposition (Oh-Witte [OWl, Prop. 3.24]). Let H be a closed, connected subgroup 
of AN in SU(2,?2;F). //, for each k G {1,2}, there is a continuous curve in H, such that 
p{h^) X ||/;,*||'^ 00 as t ^ 00, then H is a Cartan-decomposition subgroup. 
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Proof {cL Figure 8.1 and proof of Prop. 6.3). By hypothesis, there is a continuous, proper 
map $: {1,2} xR+ ^ H, such that p{^{k,t)) x . Because H C AN, we know 

that H is homeomorphic to some Euchdean space M™ (see 3.15(1)). 

Suppose, for the moment, that dimH = 1. (This will lead to a contradiction.) We know 
that p{h) h ioT h E $(1,M+). Because x h and p{h^^) x p{h) (see 8.15), we must 
also have p{h) x /i for /i e $(l,]R+)~^. There is no harm in assuming 0(1,0) = Id; then 
$(1,M+)U$(1,M+)-^ = H (because A\mH = 1), so we conclude that p(/i) x /i for all /i e H. 
This contradicts the fact that p{h) x \\hf for h e <I>(2,M+). 

We may now assume dimH > 2. Then, because H is homeomorphic to R"^, it is easy to 
extend $ to a continuous and proper map [1,2] x R+ — > H. Prom (8.18) and (8.13), 
we know that the curve p(^^'{k,t)) stays within a bounded distance from the wall L^; say 
dist [($'(/c, t)) , Lk] < C for all t. We may assume C is large enough that dist ($'(s, 1), e) < C 
for all s e [1,2]. Then an elementary homotopy argument shows that /x[$'([l,2] x M"*")] 
contains 

{ a e I dist(a, Li U L2) > C }, 
so p[^\[l,2\ X M+)] ^ Because //(//) D /i[$'([l,2] x R+)], we conclude from Theo- 
rem 2.15 that if is a Cartan-decomposition subgroup. □ 

8.20. Remark. When M-rankG = 1, the Weyl chamber has only one point at infinity. 
Thus, if H is any noncompact subgroup, then the closure of pl{H) must contain this point 
at infinity. This is why it is easy to prove that any noncompact subgroup of G is a Cartan- 
decomposition subgroup (see 2.16). 

The idea of Proposition 8.19 is that if M-rankG = 2, then the points at 00 of the Weyl 
chamber form a closed interval. If the closure of p{H) contains the two endpoints of this 
interval, then, by continuity, it must also contain all the points in between. 

Unfortunately, we have no good substitute for this proposition when R-rankG > 2. The 
points at cxo of form a closed disk (topologically speaking). It is easy to define a map / 
from one disk to another, such that the image of / contains the entire boundary sphere, 
but does not contain the interior of the disk. Thus, it does not suffice to show only that 
the closure of p{H) contains the boundary of the disk at 00; rather, one needs additional 
homotopical information to guarantee that no interior points are missed. 

8.21. Lemma. Let G — SU(2,n;F), and fix some m < n/2. Then p{S\J{l, n;¥)) and 
//(Sp(l, m; F)) are the two walls of . 

We have 

1) p{h) for he SU(1, n; F); and 

2) p\h) X \\hf for h G Sp(l,m;F). 

Proof. Let H = SU(l,n;F) or Sp(l,m;F). Then H H K is a maximal compact subgroup 
of H. From the Cartan decomposition 

H ={KnH){Ar]H){Kn H), 

and the definition of p, we conclude that p{H) = p{A n H). In the notation of (8.17), we 
see (from Definition 7.12) that H D A = L^U \ where 

^_fl if ii = SU(l,n;F); 
"I2 a H = Sp{l,m;¥). 
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Then, since A*(a~^) = /x(a) (see 8.14) and /x(a) — a ior a E we conclude that — 

is a wall of Furthermore, we have p{a) x llall*^ for a e A*(-f^) (see 8.18), so p{h) x 

hr heH (see 8.8). □ 

8.22. Corollary. // there is a continuous curve /i* — > oo m H, such that p{h^) x h* , then 
there is a compact subset C of G, such that SU(l,n;F) C CHC. 

Proof. For any (large) g e SU(l,n;F), we see from continuity (more precisely, from the 
Intermediate Value Theorem) that there exists t e such that 

11^*11 = Ibll- 

Then, by assumption and from 8.21(1), we have 

p(/i*)x/i*x^xp(^), 
so there is a compact subset C of A, such that 

/i(SU(l,n;F)) c {p{h*) | i e R+}C" C p{H)C' 

(see 8.13). Therefore 

SU(l,n;F) C /i(SU(l, n; F)) K C K p{H)C' K C K(KHK)C'K, 

as desired. □ 

The following corollary can be proved by a similar argument. (Recall that the equivalence 
relation ~ is defined in (1.15).) 

8.23. Corollary. Assume H is not compact. 

1) We have H ~ SU(1, n; F) if and only if p{h) forhe H. 

2) We have H ~ Sp(l,m;F) if and only if p{h) x \\hf for heH. 

Because of Proposition 8.19, we will often want to show that a curve /i* satisfies p(/i*) x 
for some k e {1,2}. The following lemma does half of the work. 

8.24. Lemma. Let X be a subset o/SU(2, n; F). 

1) // p{h) = 0{h) for h e X, then p{h) ^ h for h e X . 

2) // \\hf = 0{p{h)) for hex, then p{h) x \\hf for heX. 

Proof. From (8.6) and (7.10), we have 

||a|| = ai,i < ai,i 02,2 = ||p(a)|| 

and 

||p(a)|| = 01,102,2 < Oi l = Ikf 
for a e ^4+. Thus, letting o = fj.(h), and using (8.9) and (8.10), we have: 

\\h\\ >^ \\p{h)\\ < \\p{p{h))\\ >^ \\p{h)\\ 

and 

iip(/.)iix||p(M/^))ii<iiM/^)f ^ii/^ir, 

so h = 0(^p{h)) and p{h) = 0(||/i|p). The desired conclusions follow. □ 

For convenience, we record the following simple observation. (For the proof, cf. the proof 
of (8.9) and (8.10).) 

8.25. Lemma. Let 
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• ke{l,2}, 

• g & G, and 

• ^ oo be a continuous curve in H . 
Ifp{h') X then p{g-^h^g) x Wg-^h^gW". 

8D. Homogeneous functions of the same degree. The following well-known, elemen- 
tary observation is used frequently in the later sections. 

8.26. Lemma. Let V be a subspace of a finite- dimensional real vector space V , and let 
/i : y — ^ Wi and f2'- V ^ W2 be linear transformations. 

1) // f{\0) nV = {0} {or, more generally, if f{\0) n V G ^"^(O)), then there is 
a linear transformation f : Wi W2, such that f2{v) = f{fi{v)) for all v & V. 
Therefore f2 = 0{fi) on V . 

2) ///r'(0) nV'^ f^\0) n V, then /i x f2 on V. 

Proof. (1) By passing to a subspace, we may assume V = V. Then, by modding out 
fi^{0), we may assume fi is an isomorphism onto its image. Define /': fi{V) —>■ W2 by 
f'{w) = f2{fi^{w)) , and let /: Wi — >• W2 be any extension of /'. 
For V eV', we have 

11/2(^)11 = ll/(/i(^))ll< 11/11 11/1(^)11, 

so f2 = 0{f,). 

(2) From (1), we have /2 = and /i = 0(/2), so /i x /2. □ 

8.27. Example. Let f) be a real Lie subalgebra of a-|-n, and assume there does not exist a 
nonzero element -u of [), such that Xu = and = 0. Then there exist R-linear transforma- 
tions R, S : F"~^ F, such that rj^ = R{xu) + S{yu) for all u G f). (Similarly, x^, and 
are also functions of Furthermore, we have m x l^^l + 

The following well-known result is a generalization of the fact that all norms on a finite- 
dimensional vector space are equivalent up to a bounded factor. 

8.28. Lemma. IfV is any finite- dimensional real vector space, and fi, f2'- V ^ are two 
continuous, homogeneous functions of the same degree, such that f^\0) = f^^O) = {0}, 
then /i X /2. 

Proof. By continuity, the function /1//2 attains a non-zero minimum and a finite maximum 
on the unit sphere. Because /1//2 is homogeneous of degree zero, these values bound /1//2 
on all of y \ {0}. □ 

9. Existence of tessellations 

In this section, we show how to construct several famihes of homogeneous spaces that have 
tessellations. All of these examples are based on a method of T. Kobayashi (see 9.1) that 
generahzes Example 1.10. 

9A. The general Kulkarni-Kobayashi construction. As explained in the comments 
before Theorem 4.1, the following theorem is essentially due to T. Kobayashi. 

9.1. Theorem (Kobayashi, cf [Kbl, Thm. 4.7]). // 

• H and L are closed subgroups of G, with only finitely many connected components; 

• L acts properly on G/H; 
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• d{L) + d{H) = d{G); and 

• there is a cocompact lattice F in L, 

then G/ H has a tessellation. {Namely, V is a crystallographic group for G/ H.) 

Proof. Because F is a closed subgroup of L, we know that it acts properly on G/H (see 2.4). 
Thus, it suffices to show that T\G/H is compact. 

From Lemma 3.5, we see that there is no harm in assuming H C AN, and that there is a 
closed, connected subgroup L' of G, such that 

• L' is conjugate to a subgroup of AN, 

• d{L')+d{H) = d{G), and 

• L'G — LG, for some compact subset G of G. 

(Unfortunately, we cannot assume L C AN: we may not be able to replace L with L', 
because there may not be a cocompact lattice in L'. For example, there is not lattice in AN, 
because any group with a lattice must be unimodular [Rag, Rem. 1.9, p. 21].) 

It suffices to show that L'\G/H is compact. (Because L' C LG is compact, and T\L is 
compact, this implies that r\G/H is compact, as desired.) 

We know that L' acts properly on G/H (see 2.4), so L' x if acts properly on G, with 
quotient L'\G/H. Therefore, Lemma 3.19 implies that L'\G/H has the same homology 
as G; in particular, 

T^dimKiL'\G / H) = Ti-dimKiG). 

From the Iwasawa decomposition G = KAN, and because AN is homeomorphic to M'"^*^) 
(see 3.18 and 3.12), we know that G is homeomorphic to KxW''^'^\ Since R'^^'^^ is contractible, 
this implies that G is homotopy equivalent to K, so G and K have the same homology; in 
particular. 

Since 

d\Tci{L'\G / H) = dim G - dim L' - dim H 
^dimC- {d{L') + d{H)) 
^dimG-d{G) 
= dim{KAN) - dim{AN) 
— dim K, 

this implies that the top-dimensional homology of the manifold L'\G/H is nontrivial. There- 
fore L'\G/H is compact [Dol, Cor. 8.3.4], as desired. □ 

Our results for G — SU(2, 2m; F) are based on the following special case of the theorem. 
The converse of this corollary is proved in Section 11 (sec 11.5). 
Recall the equivalence relation ~, introduced in Notation 1.15. 

9.2. Corollary (Kobayashi [Kbl, Prop. 4.9]). Let H be a closed, connected subgroup of 
G = SU(2,2m;F). // 

• d{H) = 2qm; and 

• either H ~ SU(1, 2m; F) or H ^ Sp(l, m; F), 
then G/H has a tessellation. 
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Proof. Let L+ = SU(l,2m;F) and L_ — Sp(l,m;F). By assumption, we have H ~ L^, for 
some e G {+, — }; let L = L_£. Because n{L^) and ^{LJ) arc the two walls of (see 8.21), 
we know that L = L_e acts properly on G/L^ (see 2.17); since H ~ L^, this implies that L 
acts properly on G/H (see 2.4). Also, wc have 

d{L) + (i(i/) = 2qm + 2gm = d{G), 

(see 7.15 and 1.13), and there is a cocompact lattice in L (cf. 1.5(2)). Thus, the desired 
conclusion follows from Theorem 9.1. □ 

9B. Deformations of S0(2, 2m)/ SU(1, m) and SU(2, 2m)/ Sp(l, m). The homogeneous 
spaces described here were found by H. Oh and D. Witte [0W2, Thms. 4.1 and 4.6], [0W3, 
Thm. 1.5]. 

9.3. Notation. For any M-linear B : F"-^ F'^-^, wc define 

t G M, 



t X Tj 

\)B—{\ t xB —X 



r]e¥, 

X £ Fjuiag 



c a + n. 



We write xB, rather than Bx, because x is a row vector. 

It is easy to see, using, for instance, the formula for the bracket in (7.22), that if 

(9.4) lm{{vB){wBy) = - lm{vw'') for every v,w e F"-^ 

then f)B is a real Lie subalgebra of o + n; we let Hb denote the corresponding connected Lie 
subgroup of AN. 

Prom (1.14), we have 

d{HB) = dim 

= dim R + dim F"-^ + dim F + dim F-.^^^ 
^l + q{n-2) + q + {q-l) 
— qn. 

9.6. Remark. Assume n = 2m. By comparing (7.14) with (9.3), we see that there is a M- 
linear map Bq : F^"*-^ ^ g^^j^ ^-^^^ ^^(^i^ ^. p ^^^^^j ^ jj^^ (^^^^ satisfies (9.4)). 

Thus, in general, Hb is a deformation of 5p(l, m; F) fl (a + n). 

9.7. Theorem (Oh- Witte [0W2, Thms. 4.1 and 4.6]). LetB: F^""'^ F^""'^ be R-linear. 
If 

• Condition {9.4) holds, and 

• xB ^ Fx, for every nonzero x G F^"^~^, 

then 

1) p{h) X \\hf for h G Hb; and 

2) SU(2, 2m;F)/ifB has a tessellation. 

Proof. (1) Given h G Hb, write h = au, with a E A and u & N. We may assume that 

^ 1 (by replacing h with h^^ if necessary (see 8.15)). It suffices to show n^n^ 
0(01,102,2 + lA(/i)|) (for then = 0{p{h)), so Lemma 8.24(2) applies). 
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Case 1. Assume a is trivial. Prom (7.18) and (9.3), we see that 

h = 0(1x^1^ + \rih\ + \y-h\), 

so 

\\h\\'^0{\xn\' + \vh\' + \y^h\'). 
Prom (7.18) and (8.5), we have 

~ReA{h) = ^{\xh\^\yh\^ - \xy^'') + {\Vh\^ + Xhyh) . 

From (9.9), we see that — |x(a;i?)'^P > for every nonzero x e F^"*~^, so Lemma 8.28 

imphes 

I I'l ^1 |2| |2 I 1 12 

\xh\ X \xh\ \yh\ - \xhyh\ ■ 
Also, because yh — —x^ (and x^j e IFimag), we have 

\r]h\'^ + XhYh = \rih\^ + \xh\^ > 0. 

Thus, 

-ReA{h)>^\xh\''+{\vh\'' + Kn, 
so = 0(Re A(/i)) = 0(A(/i)), as desired. 

Case 2. The general case. Prom Case 1, we know = 0(1 + \A{u)\). Then, because 
ll^ll < ll^llll'^ll — we have 

< alM' = 0(a?,,(l + \A{u)\)) = 0(a?,, + a?,,|A(x.)|). 

Then, since ai,i = 02,2 and A{h) — ai^ia2,2A{u) , we conclude that = O (01102,2 + 

|A(/t)|), as desired. 

(2) From (1) and 8.23(2), we see that Hb ~ Sp(l,m;F). Then, because d{HB) = q{2m) 
(see 9.5), Theorem 9.2 implies that SU {2,2m; ¥)/ Hb has a tessellation. □ 

9.8. Lemma. Let B : F"~^ — > F"~^ &e R-linear. Condition {9.4) holds if and only if either 

1) F = R; or 

2) F = C and B' G Sp(2n — 4;M), where xB' — xB and we use the natural identification 
ofC"-^ withR^"^-^. 

Proof. Case 1. Assume F = R. Because Imz = for every 2; = 0, it is obvious that (9.4) 
holds. 

Case 2. Assume F = C. If (9.4) holds, then 

lm{{v B'){wB')^) =Im((^)(^)^) 
= Im((^;S)(wS)t) 
^ -lm{{vB){wB)'^) 
— — {— Im(fw^)) 
= Im(t'w^), 

so B' is symplectic. The argument is reversible. □ 

9.9. Remark. • For F = M, the assumption that xB ^ ¥x simply requires that B have 
no real eigenvalues. 
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• For F = C, we do not know a good description of the linear transformations B that 

satisfy xB ^ Fa;, although it is easy to see that this is an open set (and not dense). A 
family of examples was constructed by H. Oh and D. Witte (see 9.10 below). 

• If n is odd, then there does not exist B : F"~^ —>■ F"~^ satisfying the assumption that 
xB ^ ¥x. For F = M, this is simply the elementary fact that a linear transformation 
on an odd-dimensional real vector space must have a real eigenvalue. For F = C, see 
Step 1.2.1 of the proof of Proposition 10.17. 

• If n is even, then, by varying B, one can obtain uncountably many pairwise non- 
conjugate subgroups Hb, such that Sl]{2, n;¥) / Hb has a tessellation. For F = M, this 
is proved in [0W3, Thm. 1.5]). For F = C, a similar argument can be apphed to the 
examples constructed in (9.10) below. 



9.10. Example (Oh- Witte [0W2, Thm. 4.6(1)]). Assume n is even, let B' e S0(n-2;R), 
such that B' has no real eigenvalue, and define an M-linear map B : C"~^ — > C"~^ by xB — 
xB'. Let us verify that B satisfies the conditions of Theorem 9.7 (for F = C). 

Let xi,X2,yi,y2 e IR""^. From the definition of B, and because B' e S0(n — 2;R), we 
have 

Im ({{xi + iX2)B) ((2/1 + iy2)B)^) = Im (((xi - tX2)B') {{yi - ^ya)^')^) 

^t{{x,B'){y,By-{x2B'){y^By) 
= i{xiyl - X2yl) 
= -Im{{xi + ix2){yi + iy2)^)- 

Suppose Bx — Xx, for some A e C. Because B e S0(2n — 4;M), we must have |A| = 1. 
Then 



B'{x + Xx) = B'x + XB'x = B'x + XBx ^ Bx + X{Xx) ^Xx + x. 

Because B' has no real eigenvalues, we know that 1 is not an eigenvalue of B', so we conclude 
that X + Xx — 0. Similarly, because —1 is not an eigenvalue of B', we see that x — Xx — 0. 
Therefore 



x=^{{x + Xx) + {x- Xx)) = ho + 0) 



0. 



9C. Deformations of SU(2, 2m)/ SU(1, 2m). These examples are new for F = C, but 
provide nothing interesting for F = M (see 9.13(1)). 



9.11. Notation. For c e (0, 1], we define 



t X Re -|- c Im I 




t E M, 

0eF, 
X e F"-2, 
xe Fi 



imag 



It is easy to see, using, for instance, the formula for the bracket in (7.22), that f)[c] is a 
real Lie subalgebra of a + n (even without the assumption that < c < 1); we let H[c] be 
the corresponding connected Lie subgroup of AN. 
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Prom (1.14), we have 

d{Hic]) = diml)[e] 

= dim R + dim F + dim F"-^ + dim Fi^ag 
= l + g + g(n-2) + (g - 1) 
= qn. 

9.13. Remark. Let 5u(l,n;F) be embedded into 5u(2,n;F) as in 7.13. 

1) If F = R, then c is irrelevant in the definition of f)[c] (because Im0 = 0); therefore 

= su(l,n;R) n (a + n). 

2) If F = C, then ^[i] = su(l, n; C) n (a + n). 

Thus, in general, i)[c] is either 5u(l, n; F) n (a + n) or a deformation of it. 

9.14. Theorem. Assume F = C, and n — 2m is even. IfcE (0, 1], then 

1) p{h) X h for h G H^^]! cind 

2) SU(2, 2m;F)/i7[c] has a tessellation. 

Proof. (1) Given h G H[c], it suffices to show that p{h) = 0{h) (see 8.24(1)). Write h = au, 
with a E A and u E N. We may assume that oi^i > 1 (by replacing h with if necessary 
(see 8.15)). 

Let g : C e C"-2 e C ^ R be the real quadratic form 

Q{4>, X, rj) = \x\'^ + 2 Re(0r/), 
and let V be the R-subspace of C © C""^ © C defined by 

V ^ < {(j),x,ri) 



0G C, 
x G C"-2, 
= Re + c Im 



Step 1. For v gV, we have Q{v) x |0p + |a;p. For {(l),x,r)) G \ {0}, we have 

Q{(j),x,7]) = \x\^ + 2Re{(f)T]) 



\x 



2 



2Re(0(Re0 + dm 0)) 



= |xr + 2(Re0)^ -2c(Im0)^ 
> 

(because c > and lm(J) is purely imaginary). Thus, the restriction of Q to y is positive 
definite, so the desired conclusion follows from Lemma 8.28. 

Step 2. We have ^ + kwP) + From (7.19) (with y — 0), we have 

ReMi,„+2 = - + ^^{(t^uTh)^ + 2Re(0„r/^) 

and 

ImMi_„+2 = x„. 

Then, from Step 1, we sec that Rcmi^„_|_2 ^ \4>u\'^ + l^ul"^, so 

^ii,n+2 - I Retii,„+2| + I Imtii,„+2| - + + |Xu|, 
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as desired. 

Step 3. Completion of the proof. From Step 2, we have 

hi,n+2 = ai,iMi,n+2 - + \4>u\Y + ai,i|x„|. 

Also, from (7.19), we have 

(0(1) if j ^ 1 and A; ^ n + 2 

O(ai(|0„| + if j = 1 and /c 7^ n + 2 

0{\(l)u\ + \xu\) if 1 and /c = n + 2. 

Thus, it is easy to see that 

p{h) = 0(ai,i|x„| + ai,i(|0„| + = 0(/ii,„+2) = 0{h), 

so the desired conclusion follows from Lemma 8.24(1). 

(2) From (1) and 8.23(1), we see that if[c] ~ SU(l,n). Then, because d{H[c]) = 2n 
(see 9.12), Theorem 9.2 implies that SU(2, 2m; ¥)/H[c] has a tessellation. □ 

9.15. Remark. Proposition 11.6 shows that if F = C, then H[c] is not conjugate to H[c'] 
unless c = c' (for c,c' G (0,1]). Thus, Theorem 9.14(2) implies that, by varying c, one 
obtains uncountably many nonconjugate subgroups H[c], such that SU(2, 2m)/if[c] has a 
tessellation. 

9D. The product of two rank-one groups. 

9.16. Proposition. Let G = Gi x G2 be the direct product of two connected, linear, almost 
simple Lie groups Gi and G2 of real rank one, with finite center, and let H be a nontrivial, 
closed, connected, proper subgroup of AN . 

The homogeneous space G/H has a tessellation if and only if, perhaps after interchanging 
Gi and G2, there is a continuous homornorphism a: AN fl Gi — AN n G2, such that 

H = { {h,a{h)) \ he ANnGi}. 
Proof. (=^) We may assume d{Gi) > d{G2) (by interchanging Gi and G2 if necessary). 

Case 1. Assume H (1 Gi ^ e and H (1 G2 7^ e. For j = 1,2, we know that H H Gj is not 
compact (see 3.15(3)), so Corollary 2.16 implies that there is a compact subset Cj of Gj, such 
that Cj{H n Gj)Gj = Gj. Then, letting C = C1C2, we have GHG = G, so Proposition 2.9 
imphes that G/H does not have a tessellation. This is a contradiction. 

Case 2. Assume H r\Gi ^ e and H r\G2 — e. From Corollaries 2.16 and 2.15, we know that 

there is a compact subset C of An Gi, such that /x(Gi) C n{H)C. Therefore, Corollary 4.2 
(with Gi in the place of Hi) implies d{H) > d{Gi) = dim(Gi fl AN). Then, because 
H n G2 = e (and H C AN), we conclude that H is the graph of a homomorphism from 
Gi n AN to G2 n AN, as desired. 

Case 3. Assume H n Gi = e. From Corollary 4.12, we know that dim if > d{G2)- Then, 
since H (1 Gi — e, we conclude that H is the graph of a homomorphism from G2 n AN to 
Gi n AN. Interchanging Gi and G2 yields the desired conclusion. 

{<=) We verify the hypotheses of Theorem 9.1, with G2 in the role of L. 

Let H be the image of H under the natural homomorphism G G/G2. Because H C AN, 
we know that H is closed (see 3.15(1)). It is well known (and follows easily from (2.3)) that 
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any closed subgroup acts properly on the ambient group, so this implies that H acts properly 
on G/G2- From the definition of H, we have H r\ G2 = e, so wc conclude that H H acts 
properly on G/G2] equivalently, G2 acts properly on G/ H (cf. 2.3). 

Because AN = {AN fl Gi) x {AN n G2), we have d{G) = d{Gi) + d{G2). Also, we have 
d{H) = dim if (see 1.14) and, from the definition of H, we have dim if = dim{AN fl Gi) — 
d{Gi). Therefore 

d{H) + d{G2) = d{Gi) + d{G2) = d{G). 
There is a cocompact lattice in G2 (cf. 1.5(2)). 

So Theorem 9.1 implies that G/H has a tessellation. □ 

9E. T. Kobayashi's examples of higher real rank. T. Kobayashi observed that, besides 
the examples with G = SO(2,2n) or SU(2,2n) (see 1.10), Theorem 9.1 can also be used to 
construct tessellations of some homogeneous spaces G/H in which G and H are simple 
Lie groups with IR-rankG > 2. He found one pair of infinite families, and several isolated 
examples. 

9.17. Theorem (Kobayashi [Kb5, Cor. 5.6]). Each of the following homogeneous spaces has 
a tessellation: 

1) SO(4,4n)/Sp(l,n); 

2) SO(4,4n)/SO(3,4n); 

3) SO(8,8)/SO(8,7); 

4) SO(8,8)/Spin(8, 1); 

5) SO(4,4)/SO(4,l); 

6) SO(4,4)/Spin(4,3); 

7) SO(4,3)/SO(4,l); 

8) SO(4,3)/G'2(2). 

It would be very interesting to find other examples of simple Lie groups G with reductive 
subgroups H and L that satisfy the hypotheses of Theorem 9.1. 

9.18. Remark. Let G = SO(4,4n) and H' = Sp(l,n) n AN. From 9.17(1), we know that 
G/H has a tessellation. H. Oh and D. Witte [0W2, Thm. 4.6(2)] pointed out that the 
deformations G/Hb (where Hb is as in Theorem 9.7, with F = C) also have tessellations, 
but it is not known whether there are other deformations oiG/H' that also have tessellations. 

It does not seem to be known whether the other examples in Theorem 9.17 lead to non- 
trivial deformations, after intersecting H with AN. 

10. Large subgroups of S0(2,n) and SU(2,n) 

This section presents a short proof of the results we need from [OWl] and [IW]. Those 
papers provide an approximate calculation of fJ>{H), for every closed, connected subgroup H 
of S0(2, n) or SU(2, n), respectively, but here we consider only subgroups of large dimension. 
Also, we do not need a complete description of the entire set ^{H)\ we are only interested in 
whether or not there is a curve /i*, such that p(/i*) x for some k e {1,2}. The main 

results of this section are Theorem 10.19 (for k = 1) and Theorem 10.26 (for k — 2). They 
give a sharp upper bound on d{H), for subgroups H that fail to contain such a curve, and, 
if n is even, also provide a fairly explicit description of all the subgroups of AN that attain 
the bound. 
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Because of the limited scope of this section, the proof here is shorter than the previous 

work, and we are able to give a fairly unified treatment of the two groups S0(2,n) and 
SU(2,n). The arguments are elementary, but they involve case-by-case analysis and a lot of 
details, so they are not pleasant to read. 

10.1. Standing assumptions. Throughout this section: 

1) We use the notation of §7. (In particular, F = M or C, and q = dimaF.) 

2) SU(2,n;F). 

3) n > 3. 

4) if is a closed, connected subgroup of AN that is compatible with A (see 10.2), so 
dim if = d{H) (see 1.14). 

5) U = HnN. (Note that U is connected (see 3.15(2)).) 

6) u^=o = {iteu|0„ = O}. 

7) We use the notation of §8. (In particular, ||p(/i)|| is defined in (8.4) and A(/i) is defined 
in (8.5).) 

8) Except in Subsection lOA, H is compatible with A (see 10.2). 

lOA. Subgroups compatible with A. Recall that the Real Jordan Decomposition of an 
element of G is defined in (3.2); any element g of AN has a Real Jordan Decomposition 
g — au (with c trivial). If a is an element of A, rather than only conjugate to an element 
of A, we could say that g is "compatible with A." We now define a similar, useful notion 
for subgroups of AN. Lemma 10.4 shows there is usually no loss of generality in assuming 
that H is compatible with A, and Lemma 10.9 shows that the compatible subgroups can be 
described fairly explicitly. 

10.2. Definition [OWl, Defn. 2.2]. Let us say that H is compatible with A if H G TUCn{T), 
where T ^An (HN), U ^ H nN, and Cn{T) denotes the centrahzer of T in N. 

In preparation for the proofs of the main results, let us state a lemma that records a few 
of the nice properties of Jordan components. 

10.3. Lemma (cf. [Hml, Lem. 15.3, p. 99]). Let g — auc he the Real Jordan Decomposition 

of an element g ofG. Then 

1) the Real Jordan Decomposition of Ad g is Adg = (Ad a) (Ad h) (Ad c); and 

2) a, u, and c all belong to the Zariski closure of (g). 
Therefore: 

a) a, u, and c each normalize any connected subgroup of G that is normalized by g; and 

b) if (Ad g)v & v + W , for some v & Q and some (Ad g)-invariant subspace W of Q, then 

{Kda)v, {Kdu)v, and (Adc)v all belong to v -\- W . 

10.4. Lemma [OWl, Lem. 2.3]. H is conjugate, via an element of N, to a subgroup that 
is compatible with A. 

Idea of proof . Case 1. Assume, for the Real Jordan Decomposition h = au of each element h 
of H, that a and u belong to H. Let T be a maximal split torus of H. (Recall that a split 
torus is a subgroup consisting entirely of hyperbohc elements.) Then T is contained in some 
maximal split torus of G, that is, in some subgroup of G conjugate to A; replacing if by a 
conjugate, we may assume T (Z A. In other words, we now know that if n ^4 is a maximal 
split torus of H. 
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Given h E H, we have the Real Jordan Decomposition h — au. By assumption, a E H; 
thus, a belongs to some maximal split torus T' of H. A fundamental result of the theory 
of solvable algebraic groups implies that all maximal split tori of H arc conjugate via an 
element of HdN [BT, Thm. 4.21], so there is some x E H n N, such that x~^ax E A. Then 
{T,x~^ax), being a subgroup of A, is a split torus. Thus, the maximality of T implies that 
x~^ax E T; let t — x~^ax. Then 

h^au^ xtx~^u = t{r^xt)x-^u E T{H n N). 

Since h E H is arbitrary, we conclude that 

(10.5) H^T{Hr]N), 
so H is compatible with A. 

Case 2. The general case. Let 

H = {a,u \ au = ua E H, a hyperbolic, u unipotent ) 

be the subgroup of AN generated by the Jordan Components of the elements of H. (Of 
course, since every element of H has a Jordan Decomposition, we have H G H.) Then 
Case 1 applies to H, so, replacing H by a. conjugate, we may assume H — TU, where 
T = n A_and 17 = n (see 10.5). 

Because H normalizes H (see 10.3(a)), we know that Adch acts as the identity on [)/f), 
for all h E H. Hence, Lemma 10.3(b) implies that Adah acts as the identity on f)/(), for 
all h eH; therefore [H, H] C H. Also, we have [H, H] C [AN, AN] C N. Thus, letting 
U = Hr\N,we have 

(10.6) [H,H] C H nN = U. 

Because T G A and u is (AdGr(T))-invariant, the adjoint action of T on u is completely 
reducible, so (10.6) implies that there is a subspace c of u, such that [T, c] = and u + c = u. 
Therefore, U %(T) = U, so 

(10.7) H = TU = TUCjjiT) C TUCn(T). 

Let tt: AN — > yl be the projection with kernel A^, and let T = t^{H). Then 

T = tt{H) C 7r(H) = T, 

so Cn{T) D Cn{T). For any h E H, wc know, from (10.7), that there exist t E T, u E U 
and c E Cn{T), such that h = tuc. Because uc E N, we must have t = Ti{h) E T and, 
because CniT) D Civ(T), we have c E Cn{T). Therefore, h E TUCn{T). We conclude that 
H C TUCnIt), so H is compatible with A. □ 

The preceding proposition shows that H is conjugate to a subgroup H' that is compatible 
with A. The subgroup H' is usually not unique, however. The following lemma provides one 
way to change H', often to an even better subgroup. 

10.8. Lemma. Assume that H is compatible with A, and let T — Ar\{HN). IfuE Cn{T), 
then u~^Hu is compatible with A. 

Proof. Let H' = u-^Hu, T' ^ Ar\ (H'N), and U' ^ H' f] N. Because u centrahzes T, we 
have 

u-^Tu - T. 
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Also, because u E N, and N is normal, we have u ^HuN — HN, so 

u-^Tu = T = A n {HN) ^Ar\ (u-^HuN) = A n (H'N) = T'. 

Since u E N, we have u~^Nu = N, so 

u-^Uu = u-\H n N)u = (u-^Hu) n (w-^iV-u) = H'nN = U', 

and 
Thus, 

//' = u-^Hu C ii-^rC/Cjv(T)M = (m-^Tm) {u-^Uu) {u-^Cn{T)u) = T'U'Cn{T'), 
as desired. □ 

10.9. Lemma [OWl, Lem. 2.4]. If H is compatible with A, then either 

1) H ^(Hr]A)x (HnN); or 

2) there is a positive root uj, a nontrivial group homomorphism ip: kercu — > Nt^N2uj, and 
a closed, connected subgroup U of N , such that 

(a) H — { a ■0(a) | a £ kero; } U; 

(b) U is normalized by both kero; and %l){ke,ruj); and 

(c) U n '^(kera;) = e. 

Proof. Because H is compatible with A, we have H C TUCn{T), where T = An (HN) 
and U = H n N. We may assume that H ^ TU, for otherwise (1) holds. Therefore 
CjsiiT) 7^ e. Because n is a sum of root spaces, this implies that there is a positive root a;, 
such that T C kero;. Because R-rankG = 2, we have dim(kerc<;) = 1, so we must have 
T = kera; (otherwise we would have T — e, so H — U — TU] hence (1) holds). Therefore, 
Cn{T) = U^U2^. 

Because U C H C. TUCn{T), we have H = U[H n {TCn{T))], so there is a nontrivial 
one-parameter subgroup {a;*} in i7 fl {TCn{T)^ that is not contained in U. Because T 
centralizes Cn(T), we may write = a*M* where {a*} is a one-parameter subgroup of T 
and {u*} is a one-parameter subgroup of C:v(T). Furthermore, this decomposition is unique, 
because T fl Cn(T) = e. (In fact, = a*M* is the Real Jordan Decomposition of x*.) Define 
-0: kero; ^ U^U2uj by 0(a*) = for aU t e M. 

(2a) For all i e M, we have a^il}{a^) = a^u^ = e i?, which estabhshes one inclusion of (2a). 
The other will follow if we show that dimi? — dimC/ = 1, so suppose dim if — dimC/ > 2. 
Then Lemma 10.11(1) implies that A C if, so it follows from Lemma 3.21 (with T — A and 
a; = 0) that H = A k {H fl N), contradicting our assumption that H ^ TU. 

(2b) Because a;* e H, we know that each of a* and m* normalizes if (see 10.3(a)). Being 
in AN, they also normalize N. Therefore, they normalize H O N — U. 

(2c) Suppose U n-?/'(kercj) 7^ e. Because the intersection U nipiker u) is connected (see 2b 
and 3.15(2)), and dim(kera;) = 1, we must have iplkeruj) C U. Therefore a* = x^u~^ G 
HU = H, so T G H. This contradicts our assumption that H ^ TU. □ 

10.10. Corollary. If H is compatible with A, then A fl (HN) normalizes H (1 N. 



10.11. Lemma [OWl, Lem. 2.8]. i/ dim (ii/(ii n AT)) > dimA, then 
1) H contains a conjugate of A; and 
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2) H is a Cartan-decomposition subgroup. 

Proof. (1) Let tt: AN Ahe the projection with kernel N, and let H be the Zariski closure 
of H. Prom the structure theory of solvable algebraic groups [Br2, Thm. 10.6(4), pp. 137- 
138], we know that H = T k U is the semidirect product of a torus T and and unipotent 
subgroup U <Z N . Replacing i7 by a conjugate under N , we may assume that T <Z A. Since 

dim A < d\ui{H/{H n N)) = dim(7r(ii')) < dim A, 

we must have 7r(if) = A, so 

A = t:{H) C t:{H) = 7r(TC/) ^TcH 

normalizes H (see 10.12). Then, since 7r{H) — A, we conclude that A C H (see 3.21). 

(2) From (1), we see that, by replacing H with a conjugate subgroup, we may assume 
A G H. Because A is a Cartan-decomposition subgroup (see 2.6), this implies H is a 
Cartan-decomposition subgroup. □ 

The following basic result was used twice in the above arguments. 

10.12. Lemma (cf. [Zm2, pf. of Thm. 3.2.5, p. 42]). If H is a closed, connected subgroup 
of G, then the Zariski closure of H normalizes H. 

IDE. Subgroups with no nearly lineEir curve. Our goal is to prove Theorem 10.19; we 
begin with some preliminary results. 

First, an observation that simplifies the calculations in some cases, by allowing us to 
assume that — 0. 

10.13. Lemma. Let u & u. If dimv{¥xu + ^yu) < 1 md Uu 7^ 0, then there is some g G N^, 
such that 

1) ■^g~^ug 0; 

2) 0(,-i«£; = and 

3) Ug-^ug — Uu- 

Proof. Because (\im.^{¥xu + Fy^) < 1 and iju 7^ 0, there is some A e F, such that Xy, — Xyu- 

Let 

• f be the element of Uq with (j)^ = —A, 

• g — exp(i>) e Na, and 

• w — g~^ug. 

From (7.23), we see that 

• Xy, = Xu + (pvUu = 0, and 

as desired. □ 

10.14. Proposition. If there does not exist a continuous curve /i* — > 00 m U, such that 

p{h^) X h^, then 

1) for every nonzero element z ofdt,, we have |?7^p + x^y^ 7^ 0; and 

2) for every element u of such that dim]F(Fa;„ -|- F?/„) = 1, we have 

^u\yu? + )/u\xu? + 2 \m{xuyir]l) ^ 0. 
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Proof of the contrapositive. (1) Suppose there is a nonzero element z of Di, with A.{z) — 0. 
Let /i* = ex.-p(tz) = Id +tz (see 7.18). We have 

_i0{t) for all J, fc, 

\0(1) if (j,A:)^{l,2}x{n + l,n + 2}. 

Then, because A(/i*) = 0, it is easy to see that p{h) x t. Also, we have /i* = Id +tz x t, so 
p(/i*) X t X /i*, as desired. 

(2) Suppose there is an element u of u^=o, such that dim]{r(Fa;u + Fy„) = 1, and 

(10.15) y^ulVu]'^ + yu\xu\^ + 2 Im(a;„2/^?7^) = 0. 

Let h = = exp{tu). 

Case 1. Assume Xu = ^- Because dimF(Fa;„ + Fy„) = 1, we must have ^ 0. Then, 
from (10.15), we know that x„ = 0. So, from (7.18), we see that 

• hj^k — 0{t) whenever (j, k) ^ (2, n + 1), and 

• hj^k — 0(1) whenever j 7^ 2 and k^n^l. 

This implies that piK) x x /i. 

Case 2. Assume yu — 0. This is similar to Case 1. (In fact, this can be obtained as a corollary 
of Case 1 by replacing H with its conjugate under the Weyl reflection corresponding to the 
root a.) 

Case 3. Assume 7^ 0. Because dimiF(Fx„ + F|/„) = 1, Lemma 10.13 implies there is some 
g G Na, such that, letting w = g^^ug, wc have </).„, = 0„ = 0, x,^, = 0, and yw = Uu^ 0. We 
show below that (10.15) is satisfied with w in the place of m, so, from Case 1, we conclude 
that p(exp(iu')) x exp(tu'). Thus, the desired conclusion follows from Lemma 8.25 (with 
k^l). 

To complete the proof, we now show that (10.15) is satisfied with w in the place of u. 
(This can be verified by direct calculation, but we give a more conceptual proof.) Because 
e Na-i multiplication by g~^ on the left performs a row operation on the first two rows 
of h] likewise, multiplication by g on the right performs a column operation on the last two 
columns of h. These operations do not change the determinant A(/i): thus 

A(exp(tw)) = A(exp(iii)). 

Prom (7.18) and the definition of A, we see that 

A(exp(iM)) = ~{\x^\^\yu\^ - \xuyi\^y + {^\yu? + yu\xu? + ^^1^^ 

Because dimF(Fx„ + Fy„) = 1, we have — la^uy^P = 0, so this simplifies to 

A(exp(tM)) = {x.,\yu\'' + yu\xu? + '2\rn{x.,yl7]l))e + 0{t'). 
Thus, (10.15) is equivalent to the condition that A(exp(t-u)) = Oit^). Then, since 

A(exp(^^/;)) = A(exp(iii)) = 0{t^), 
we conclude that (10.15) is also valid for w. □ 
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10.16. Lemma. If there does not exist a continuous curve /i* ^ oo m U, such that p{h*) x 



Furthermore, if equality holds, and F = then u = \x^=q and dim Dp, = 3. 

Proof. Case 1. Assume F = C. Because + Xz^z is a quadratic form of signature (1,3) 
on t), we know, from 10.14(1), that dim()^ < 3 = 2q' — 1. 

Thus, we may assume u/u0=o 7^ 0, so there is some w e u, such that 0^ 7^ 0. 

Subcase 1.1. Assume there exists z G ff,, such that = and rjz ^ R0u. From (7.22), we 
see that [u, z] e Of,, with xj^^^j = — Im(0u7y^) ^ and y[u,z] — rj[u,z] — 0. This contradicts 



Subcase 1.2. Assume there exists z G '0^, such that 7^ 0. From (7.22), we see that [m, 2;] is 
an element of Oj,, such that y[u,z\ = 0, and ri[u,z] = (puYz is a purely imaginary multiple of 
So Subcase 1.1 applies (with [u, z] in the place of z). 

Subcase 1.3. Assume = and rjz G for all z G D;,. From 10.14(1), we see that 

^i) n n2a+2/3 — {0}, so the assumption of this subcase implies dimt)^ < 1. Thus, 



so the desired inequality holds. 

If equality holds, then dim Of, = 1 and dimu/u<^=o = 2. Thus, we may choose z G 0^, such 
that -2 7^ 0, and u' G u, such that ]R0„ + M(/)u' = C. From the assumption of this subcase, we 
know that rjz G M0u; thus, r]z ^ Therefore, Subcase 1.1 applies, with u' in the place 

of u. 

Case 2. Assume F = M. Because dimOj, < dimna+2/3 = 1 and dimu/u^=o < dimUc = 1, the 
desired inequality holds unless Of, 7^ and u/u^=o 7^ 0. Thus, we may assume there is some 
nonzero 2; G Of, and some u & u, such that 0„ 7^ 0. 

Subcase 2.1. Assume iju = 0. We may assume \xy\'^ + 0„77u 7^ (by replacing u with u-\- z, 
if necessary). Let /i* = exp(iii). From (7.19), we see that h\ ,^_^2 ^ but 



Therefore p(/i*) = O(t^) = 0{h^), so Lemma 8.24(1) implies that p(/i*) x /i*. This is a 
contradiction. 

Subcase 2.2. Assume yu 7^ 0. Let v be the element of xip with = — (l/0„)a;„, and let 
w = exp(— v)m exp(?;). Then x^,, = (see 7.23 and 7.22). Thus, by replacing H with the 
conjugate exp(— f )if exp(f ) (see 8.25), we may assume Xu = 0. For any large real number t, 
let h = h^ he the element of exp{tu + na+2/3) that satisfies r]h = — 0/i||7//i||^/12. Then, 
from (7.17), we see that 



h^, then 



dim Of, + dimu/u^=o < 2g — 1. 



10.14(1). 



dimOi, + dimu/u^=o <l + 2 = 3 = 2g-l. 




/i 4>h l<Phyh 




1 Vh 



h 



1 



V 
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Clearly, we have h 4>Wyh\^ . 

A calculation shows that A(/i) = 0, and certain other 2x2 minors also have cancellation. 
With this in mind, it is not difficult to verify that p{h) x (pWuhl^ ^ h (see [OWl, Case 3 of 
pf. of 5.12(3 =^ 2)] for details). This is a contradiction. □ 

10.17. Proposition. If there does not exist a continuous curve /i* ^ oo m U^=q, such that 
p{h^) X then 

{q{n — 2) if n is even 
q{n — 3) if n is odd and n ^ 2> 
q — 1 if n = 3. 

Furthermore, 

a) if equality holds, and n is even, then dimf(¥xu + ^Vu) = 2, for every u G u^=o \ Oj,; 

b) if equality holds, and n = 3, then dim Sf, < q. 

Proof By passing to a subgroup, we may assume u = u^=o. Let V be the projection of u to 
+ tta+/3; then dimT^ = dimu/ dimt)f|. 

Case 1. Assume dimF(Fx„ + Fy„) = 2 for every m e u \ Df,. 

Subcase 1.1. Assume n is even. Prom Theorem 10.14(1), we know that V does not intersect 
xvp (or xia+p, either, for that matter), so 

dim V + dim xip < dim(n/3 + xia+p) — dim xvp + dim tia+z?. 

Therefore 

dimu/ diml)f, = diml^ < dimno,+^ = q{n — 2), 
as desired. (If equality holds, then we have Conclusion (a).) 

Subcase 1.2. Assume n is odd. 

Step 1.2.1. We have dimu/?);, < q{n — 3). Suppose not: then 

dimy >5(n-3) + l. 

(This will lead to a contradiction.) Let X — {x^ | v e 1/}, so X is a R-subspace of F""^. 
For each x e X, there is some v — v{x) & V , such that x^ — x] define f{x) — yv{x)- By the 
assumption of this case, we know 

Vnnp = {0}, 

so v{x) is uniquely determined by x; thus, /: X — > F*^"^ is a well-defined M-linear map. 
Also, again from the assumption of this case, we know that 

(10.18) f{x) ^ Fx for every nonzero x E X. 

Because V fl n^j = 0, we have 

dimX = dimV > q{n - 3) + 1 = dimF"'^ - (q-l). 

If F = M (that is, if g = 1), this implies X = ]R"~^, so / is defined on aU of M""^. Because 
n is odd, this implies that / has a real eigenvalue, which contradicts (10.18). 
We may now assume F = C. Let 

• E^{X X C"-2)/=, where {x,v) = {-x, -v), 

• FX be the projective space of the real vector space X, and 

• C(^,^^) = [x] e FX, for {x,v) e E, 
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SO (EX) is a vector bundle over FX. 

Define g: X ^ C"""^ by g{x) = ix. Any M-linear transformation Q: X ^ C""^ is a 
continuous function, sucli tliat Q{—x) = —Q{x) for all x G X; that is, a section of {E,()- 
Thus, Id, /, and g each define a section of {E,()- Furthermore, these three sections are 
pointwise linearly independent over M, because (10.18) implies that x, f{x), and ix are 
linearly independent over M, for every nonzero x & X. On the other hand, the theory of 
characteristic classes [MS, Prop. 4, p. 39] implies that {E,() does not have three pointwise 
M- linearly independent sections (see [IW, Lem. 8.2] for details). This is a contradiction. 

Step 1.2.2. Completion of the proof of Subcase 1.2. From Step 1.2.1, we see that the desired 
inequality holds. 

We may now assume n = 3 and dimu/Of, = q — 1. Since dimu/Of, < q{n — 3) = 0, we must 
have g = 1, so F = R. Therefore xv2a — ri2/3 = 0, so 

dim?)^ < dimna+2/3 = 

as desired. 

Case 2. Assume there is some f G u \ c)(,, such that (\\mf(¥x.^ + Fy^,) = 1. 

Subcase 2.1. Assume x^ = 0. Since v ^ we must have ^ 0. Then x^^^ 7^ for every 
2; G Of, (otherwise 10.14(2) yields a contradiction); this implies 

x„ 7^ 0, 

and 

= for every z G c)(,. 
Because 7^ 0, we know that F 7^ M; so 

F = C. 

Since x^ = for every z G ?)[,, but Of, fl 1X2/3 — (see 10.14(1)), we must have rj^ ^ for every 
2; G Of,. Therefore 

dim Of, < dimna+2/3 = q = 2. 
Let p: V ^ tta+/3 be the natural projection. Note that 

dimkerp = 1. 

(If v' G u, with x^/ = 0, then there is some t G M, such that Xy'^t^ = x„/ + tx^ = 0. We 
also have x^/^tv = 0, so, from 10.14(2), we see that v' + tv G Of,. Thus v' G M.v + Of,. So 
kerp = {Rv + Of,)/Of, is 1-dimensional.) 

Because x^, = for every z G Of,, and u is a Lie algebra, we see, from (7.22), that p{V) 
must be a totally isotropic subspace for the symplectic form ilm{xx^), so 

dimp{V) < - dimna+/3 ~ n — 2. 

Therefore 

dimu/Of, = dim = dimp(V^) + dimkerp < {n 

This completes the proof if n 7^ 3: 

• If n is even, then, because n > 4, we have n — 1 < 2{n 

• If n > 3 is odd, then n > 5, so n — 1 < 2{n — 3) = q{n 



-2) + l = n-l. 

-2) = g(n-2). 
-3). 
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Now let n — 3, and suppose dimy = 2. (This will lead to a contradiction.) Because 
equality is attained in the proof above, we must have dimp{V) — n — 2 — 1, so there exists 
w eu with Xyj ^ 0. For i e M, let = w + tv. Then 




+x^oo as i — > oo 
— x„oo as i — > — oo. 



Thus, this expression changes sign, so it must vanish for some t. On the other hand, since 
n = 3, we have dimc(Cx + Cy) < 1 for every x,y e C"-^ = C. Thus 10.14(2) yields a 
contradiction. 

Subcase 2.2. Assume = 0. This is similar to Subcase 2.1. (In fact, this can be obtained 
as a corollary of Subcase 2.1 by replacing H with its conjugate under the Weyl reflection 
corresponding to the root a.) 

Subcase 2.3. Assume y^ ^ 0. Because dimir(Fa;^, + Fy^,) = 1, Lemma 10.13 implies there is 
some g e A'q,, such that, letting w = g~^vg, we have (pw = 4>u = 0, Xy^ = 0, and yw = Uv ^ 0. 
There is no harm in replacing H with g~^Hg (see 8.25). Then Subcase 2.1 apphes (with w 
in the place of t;). □ 



10.19. Theorem. Recall that Assumptions 10.1 are in effect. 

If there does not exist a continuous curve ^ oo in H, such that p{h^) x h^, then 



(10.20) dimH< 




if n is even 
if n is odd. 

Furthermore, if equality holds, and n is even, then 

1) {) = (ker a) K u; 

2) = for every u G u; 

3) dimF(Fa;„ + F|/u) = 2, for every m e u \ c)(,; 

4) + x^y^ 7^ for every nonzero z e D^; 

5) dimu/Of, = q{n — 2); and 

6) dim ()^ = 2g — 1. 




if n is even 
m— I qin — 3) if n > 5 is odd 



Proof. Let 



if n = 3. 

Prom Lemmas 10.11(1) and 10.16, and Proposition 10.17, we have 

dim H < dim l^/u + (dim u/u0=o + dim ?)(,)+ dim u<^=o /f \) 
(10.21) <l + (25-l)+m 

— 2q-\-m. 

This implies the desired inequality, unless n = 3, F = C, and we have equality in both 
Lemma 10.16 and Proposition 10.17. This is impossible, because equality in Lemma 10.16 
requires dim()(, = 3, but Proposition 10.17(b) implies dim()(, < 2. 
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Assumption. In the remainder of the proof, we assume that equality holds in (10.20), and 
that n is even. Proposition 10.14(1) implies (4). 

Case 1. Assume F = C. Because equality holds, Lemma 10.16 implies (2) and (6). Then 
Proposition 10.17(a) implies (3) (because u = u<^=o)- Since u = u<^=o (see 2) and equality 
holds in (10.20), we have 

(10.22) dimu/()(, = dimu^=o/''() — m — q{n — 2) 
and 

(10.23) dimDf, = dimu/u0=o + dimDf, = 2g — 1, 

so (5) and (6) hold. 

Let T — An (HN). Corollary 10.10 imphes that T normahzes u, so, from (3) and 
Lemma 3.21, we sec that T C kcra. On the other hand, dimT = diml)/u, so, from equality 
in (10.21), we conclude that dimT = 1. Therefore T = kera. 

Suppose ip: kera — > Na is any continuous group homomorphism, such that ip{keTa) 
normalizes U. Prom (3) and (7.22), we see that Nn^{U) — e, so ip must be trivial. This 
imphes that 10.9(2) cannot apply here, so 10.9(1) yields (1). 

Case 2. Assume F = R. Proposition 10.17(a) implies that dimF(Fx„ + Fy^) = 2 for every 
u e U0=o \ f ()• 

Suppose (2) is false. Then there is some -u G u, such that ^ 0. Also, because 
dimu0=o/ff) = m > 0, we may fix some v G u^=o \ Then, letting w = [u,v], we 
see, from (7.22), that Uw — and 7^ 0, so dimF(Fxt„ + Fy^) = 1. This contradicts the 
conclusion of the preceding paragraph. 

Conclusion (3) follows from (2) and 10.17(a). 

Conclusion (1) can be established by arguing as in the last two paragraphs of Case 1. 
Equations (10.22) and (10.23) establish (5) and (6). □ 

IOC. Subgroups with no nearly quadratic curve. Our goal is to prove Theorem 10.26; 

wc start with two preliminary results. 

10.24. Lemma. If there does not exist a continuous curve /i* ^ 00 m U, such that p{h}) x 
then 

1) for every element u of we have dimF(Fa;„ + F|/„) < 1; 

2) for every element z ofD\^, we have \r]z\'^ -^y^z^z — 0; and 

3) for every element u of u, such that 0^ ^ 0, yu — 0, and = 0, we have \xu\^ + 

Proof of the contrapositive. (1) Suppose there is an element u of u^=o, such that dimF(Fxu + 
Fyu) — 2. Let /i* = exp(tti). Then, from (7.18), we see that — 0{t^). Furthermore, 

A{h^) = det - 'rf^, -^;}\"fJt^ = \\\^u\\yu? - Kyi\Y + 0{t% 

Because dimF(Fxu + Fy„) = 2, we have > so \xuyl\^ 7^ 0; therefore 

A(/i*) X t\ so 

\\hr = Oit') = 0{A{h'))=0{p{h')), 
so Lemma 8.24(2) implies that p(/i*) x as desired. 
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(2) Suppose there is an element z oiX>^, such that \r]z\'^ + Xzyz ^ 0; in other words, we have 
A(z) ^ 0. Let U = exp(iz) = Id+t^ (see 7.18). Then U = and 

t'x A(z)i' = A(/i*) = 0(p(/i*)), 

so 

\\h'f = 0{f)=0[p{h')), 

so Lemma 8.24(2) imphes that p(/i*) x as desired. 

(3) Suppose there is an element u of u, such that 0^ 7^ 0, = 0, y„ = 0, and + 
2Re(0„r/;^) = 0. Let U — exp{tu). From (7.19), we see that /i* = Id+tu (note that, because 
l^up + 2 Re(0„r7;;) = 0, we have Rehl^_^_2 = 0). Then /i* = 0(i) and 



det ( , fi'"+2 



det 



t0„ * 

-^01 



So \\hY = 0(i2) = 0{p{h^)). Thus, Lemma 8.24(2) implies that p{h*) x as desired. 

□ 

The following lemma obtains a dimension bound from Condition 10.24(1). 

10.25. Lemma. If V is a ^-suhspace of F"-^ e F"-^ such that dimF(Fx + Fy) < 1 for 
every (x, y) e V , then either 

1) dim V < q{n — 2); or 

2) n = 3 and dimK < 2q. 

Proof. Because dim^F"^^ = g(n — 2), we may assume that there exist nonzero xq, j/o G F"^^, 
such that (a;o,0) e V and (0,|/o) € V (otherwise, the projection to one of the factors of 
]p"-2 is injective when restricted to V, so (1) holds). Then (a;o,|/o) G l^, so, by 
assumption, we have dim]{r(Fxo + Fyo) < 1- Because and yo are nonzero, this implies 
Fxo = Fyo- 

S'tep 1. For all {x,y) G V, we have y G ¥xq. We may assume y ^ (otherwise the desired 
conclusion is obvious). Then, since dimF(Fa; + ¥y) < 1, we conclude that x G ¥y. Similarly, 
because 

{x + xo, y) = (x, y) + (xo, 0) G F + F = 
we must have x + G Fy. Therefore 

Xo = (x + Xo) - X G Fy - Fy = Fy. 
Since xo 7^ 0, this implies Fxo = Fy, so y G Fxo, as desired. 

^'tep ^. We have V C Fyo Fxo- Given (a;,y) G V, Step 1 asserts that y G Fxo. By 
symmetry (interchanging the two factors of F""^ ® F"~^), we must also have x G Fyo. So 
(x, y) G Fyo ® Fxo, as desired. 

.S'tep 3. Completion of the proof. From Step 2, we have 

dimy < dimK(Fyo ® Fxo) = 'iq. 
If n > 4, then (1) holds; otherwise, (2) holds. □ 

10.26. Theorem. Recall that Assumptions 10.1 are in effect. 

If there does not exist a continuous curve /i* — > 00 m H , such that p{h^) x then 
dim if < qn. 

Furthermore, if equality holds, then H is of the form H = T x U , where 
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1) T = ker/3, 

2) u = ((n„ + n„+/3 + n„+2/3) n u) + 1x2^+2/3; and 

3) \xu\'^ + 2 Re{(f)Jh) 7^ for every it e u \ n2a+2/3- 

Proof. Note that 

dim ^/u < 1 

(see 10.11(1)) and 

dimu/u<^=o ^ dimno; = q. 

Step 1. We have dimu<^=o/f fi < — 2). Suppose not. Let V be the projection of u^=q to 
+ tta+/3- We have 

dimV^ = dimu0=o/f f) > ^(^ ~ 2), 
and, for every u G 11,^=0 with Xu 7^ 0, we have dimF(F,x„ + Fy„) < 1 (sec 10.24(1)), so 
Lemma 10.25 imphes that n = 3. Therefore dimn/3 = dimria+g = q. Then, because 
dimy > q{n — 2) = q, we know that V Hxip ^ and V fl Ua+p 7^ 0; thus, there exist 
u,v & U0=o, such that 

• x„ = 0, y„ 7^ 0; and 

• a;„ 7^ 0, = 0. 

Therefore [u.v] is a nonzero element of 1x^+2/3 (see 7.22), so A([m, v]) 7^ 0. This contradicts 
Lemma 10.24(2). 

Step 2. We have dimc)(, < g — 1. Suppose not: then, because dimn2a+2,3 = 9 ~ 1) there is 
some M G \ n2a+2/3, and, because dim 1x2/3 = 9 ~ 1) there is some nonzero v G Of,, such that 
= 0. We must have 77^, = (otherwise 10.24(2) yields a contradiction); thus v G xi2a+2i3- 
We must have 7^ (otherwise 10.24(2) yields a contradiction). Thus, we see that 

\r}\^ + Xu+tvyu+tv = \t1u\^ + (x« + tx^)(tyu) 
is nonconstant as a function of i G R, so 10.24(2) yields a contradiction. 
Step 3. The desired inequality. We have 

dim [) < dim f)/u + dimu/U(^=o + dimu0=o/f i) + dim Of, 
<l + q + q{n-2) + {q-l) 

as desired. 

Assumption. In the remainder of the proof, we assume that dimi? = qn. We must have 
equality throughout the preceding paragraphs. 

Step 4- We have V C xXa+p. Suppose not: then there is some v G u^=o, such that y^ 7^ 0. 
Let M G u \ u<^=o and w = [u,v]. Then, from (7.22), we see that yw — and 7^ 0, and 
that [v, w] G no,+2/3 + ^2a+2i3- Ftom 10.24(1), we have Xy G ¥yy and Xy+2 £ ^Uv+w — ^Uv, so 

Xyj Xy-^yj Xy G Fl/|; -I^^/ll -^^/f * 

Therefore Xyjyj^ 7^ 0, so 7^ (see 7.22), so 10.24(2) yields a contradiction, 
^tep 5. We have Oj, = Vi2a+2i3- From Step 4, together with the fact that 

diml^ = dimu</,=o/()(, = q{n — 2) — dimna+^j. 



66 ALESSANDRA lOZZI AND DAVE WITTE 

we conclude that V — n^+p- Therefore, 

U0=o + C) = y + t) = Wa+p + 9, 

SO 

= [u<^=o + f,U<^=o + f] 

= [xXa+fS + f , na+/3 + f] 

Because dimc)(, = q — 1 = dim 1X20+2/3, we must have Of, = n2a+2/3- 

Step 6. We have Of, = n2a+2(3- Let T = {HN) H A be the projection of H to A. Then there 
exists a e a + a + 2/3}, such that T — ker(Q; — a), and, in the notation of Lemma 3.21, 
we have 

u= (unn=") + (unn^"). 
Because T normahzes u (see 10.10), we know, from Lemma 3.21, that u = (unn^") + (unn^"). 
Since dimu/u0=o = we know that ufln"" projects nontrivially (in fact, surjectively) to n^. 
On the other hand, we know that u n Uq, = (otherwise 10.24(3) yields a contradiction). 
Therefore 7^ n^, so there must be a positive root a ^ a, such that a\T — Q;|r- Then 
T C ker(Q; — a); since dimT = dmiH/U — 1, we must have T — ker(Q; — a). 
Because u n n^" C u<^=o, we have 

dim(u n n ) > dim = dim — ^— -j— = dim = q. 

u<^=onn=" u<^=o/(unn^") u^=o 

Then, since u n Uq, = 0, we must have 

dimn"" > dim(u fl n"") + dimn^ > q + q = 2q. 

By inspection, we see that this implies a ^ {2a, 2(3}, so we conclude that a e {/?, a + (3,a + 
2(3}, as desired. 

Step 7. We have a G {a + (3,a + 2(3}, T = kerP, and n"" = n„ + Ua+p + na+2f3- Since 
ker (3 = ker 2(3, it suffices to show a ^ (3. Thus, let us suppose a = (3. (This will lead to 
a contradiction.) We have = ti^ + (and recall that u n tIq, = {0}), so there is some 
u & u, such that 0^ 7^ and y„ ^ 0. Because V — Ua+p, we have 

dim{ V eV \xy e ¥yu } = g > dimFimag, 

so there is some v e u<^=o, such that ^ Xy & Fy^ and y,, = 0. Then [u, v] e na+2/3 + W2a+2^, 
with rj[u,v] 7^ (see 7.22), so 10.24(2) yields a contradiction. 

Step 8. We have H — {H r\ A) x {H D N). Suppose not: because T = kcr/3, we conclude 
that there is some nonzero w & + 1x2/3, such that w normalizes u (see 10.9). 

U yw 7^ 0, then, because V = xia+p, there is some v e U0=o, such that G ¥xy and = 0. 
Then [w,v] G nQ+2/3 + Ti2a+2/3, with rj^yj^y] ^ (see 7.22), so 10.24(2) yields a contradiction. 

If yyj = 0, then, since w ^ 0, we must have y^ 7^ 0. There is some v & u with (f)y ^ 0. 
Then [w,v] e na+2p + W2a+2/3, with rj[yj^y] ^ (see 7.22), so 10.24(2) yields a contradiction. 

Step 9. Completion of the proof. (1) Prom Step 8, We know that H — T K U, and, from 
Step 7, that T = ker (3. 
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(2) Since n=°^ = na + n„+/3 + na+2/3, it suffices to show uHn^" = n2a+20' given v e uHn^", 
we wish to show v G 1x20+2/3- Because V = na+/3, we know that y.y = 0. Thus, all that 
remains is to show that y^, = 0. If not, then choosing u & u with 0^ ^ 0, we see that 
Vlu,v] 7^ (see 7.22). So 10.24(2) yields a contradiction. 

(3) From Lemma 10.24(3), we know that |a;„p+2 Re{(j)uf}^) ^ for every u e u\n2a+2/3- □ 

11. HOMOGENEOUS SPACES OF S0(2, n) AND SU(2, n) 

This section proves two main results. Both assume that G is either S0(2,n) or SU(2,n). 

1) Theorem 11.1 shows that if n is odd, and one or two specific homogeneous spaces of G 
do not have tessellations, then no interesting homogeneous space of G has a tessellation. 

2) Theorem 11.2 shows that if n is even, then certain deformations of the examples found 
by R. Kulkarni and T. Kobayashi (see 1.10) are essentially the only interesting homo- 
geneous spaces of G that have tessellations. 

The classification results of §10 (specifically. Theorems 10.26 and 10.19) play a crucial role 

in the proofs. 

We use the notation SU(2,n;F) of Section 7, to provide a fairly unified treatment of 
S0(2,n) and SU(2,n). 

11.1. Theorem. Assume G = SU(2,2m + 1;F) with m > 1, and let H be any closed, 
connected subgroup of G, such that neither H nor G/H is compact. 
If Conjecture 1.17 is true, then G/H does not have a tessellation. 

Proof. Assume Conjecture 1.17 is true, and suppose F is a crystallographic group for G/H. 
(This will lead to a contradiction.) Let 

Hi = SU(1, 2m + 1; F) and H2 = Sp(l, m; F) 

(see 7.12). From (7.15), we have d{Hi) = q{2m + 1) and d{H2) = q{2m), where q = dimjjF. 
We may assume that H C AN (see 3.5), and that H is compatible with A (see 10.4). 

Because H is not a Cartan-decomposition subgroup (see 2.9), the contrapositive of Propo- 
sition 8.19 implies, for some A; G {1, 2}, that there does not exist a continuous curve /i* — 00 
in H, such that p(/i*) x ||/?/||'^. Therefore, cither Theorem 10.19 (if A; = 1) or Theorem 10.26 
(if A; = 2) implies that d{H) < q{2m + 1) = d{Hi). 

We consider two cases. 

Case 1. Assume that F acts properly dis continuously on G/Hi. Theorem 4.1(2) (combined 
with the fact that d{H) < d{Hi)) implies that G/Hi has a tessellation. This contradicts 
either Theorem 1.18 (if F = M) or Conjecture 1.17c (if F = C). 

Case 2. Assume that F does not act properly dis continuously on G/H^. From Lemma 8.21, 
wc know that n{Hi) and li{H2) arc the two walls of A"*", so Corollary 4.11 (combined with the 
assumption of this case) implies that F acts properly discontinuously on G/H2. Therefore, 
since Conjecture 1.17ab asserts that G/H2 does not have a tessellation, the contrapositive of 
Theorem 4.1(2) (with H2 in the role of Hi) implies that d{H) > d{H2) = q{2m). Hence, the 
contrapositive of Theorem 10.19 implies there is a continuous curve /i* — > cxo in H, such that 
p(/i*) X /i*. Thus, there is a compact subset G of G, such that Hi C GHG (see 8.22). Since 
F acts properly discontinuously on G/H, this implies that F acts properly discontinuously 
on G/Hi (see 2.4). This contradicts the assumption of this case. □ 
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11.2. Theorem. Assume G — SU(2,2m;F) with m>2, and let H he a closed, connected, 
nontrivial, proper subgroup of AN. 

The homogeneous space G/H has a tessellation if and only if either 

1) there is an ^.-linear m,ap B : F"^^ —> F"~^, such that 

(a) Im(^{v B){wBy^ = — lm{vw'') for every v,w & ]F"-2 (^ggg 9.8), and 

(b) xB ^ Fa;, for every nonzero x e F"~^ {see 9.9), and 

(c) H is conjugate to Hb {see 9.3 and 9.6); or 

2) ¥ = R and H is conjugate to SU(1, 2m; R) n AN {see 7.12); or 

3) F = C and there exists c e (0, l], such that H is conjugate to H^^] {see 9.11). 

Proof. (4=) See (1) Theorem 9.7(2), (2) Theorem 9.2 (and 7.15), or (3) Theorem 9.14(2). 

(=^) Let n = 2m, so G = SU(2,n;F). By combining Remark 1.14, Corollary 4.12, 
Lemma 8.21, and Remark 7.15, we see that 

dim H = d{H) >min{rf(SU(l,n;F)),d(Sp(l,m;F)) } = qn. 

Also, we may assume H is compatible with A (see 10.4). Because H is not a Cartan- 
decomposition subgroup (see 2.9), Proposition 8.19 implies that one of the following two 
cases applies. 

Case 1. Assume there does not exist a continuous curve /i* ^ oo m H, such that p{h^) x /i*. 
Since d\m.H > qn, Theorem 10.19 implies that dim if = qn, and that H is of the form 
H xU (with U C N), where 

i) T = ker a; 

ii) 0u = for every -u G u; 

iii) dim][r(Fx„ + F?/„) = 2, for every m e u \ c)(,; 

iv) I^^P + ^zYz 7^ for every nonzero z G Of,; 

v) dimu/t)[, = q{n — 2); and 

vi) dim Of, — 2q — 1. 

Step 1.1. We may assume that Of, = { z G | = — y^}. Because dim Of, = 2q — 1 
(see vi), it suffices to show that = — for all z G Of,. This is trivially true if F = M, as 
x^jYz £ Fimag = {0} in this case. Thus, we assume F = C. 

For any -2 G Of, with rjz =)/z = 0, we know, from (iv), that z = 0; therefore. Lemma 8.26(1) 
implies there exist R-linear maps R: C — > and S: iR iR, such that Xz = R{f]z) + S{yz) 
for all z G Of,. More concretely, we may say that there exist A G C and c G R, such that 
Xz = Im(Ar7^) + cy^ for all z G Of,. 

Let V be the element of with = A/2, and let H* = exp{—v)Hexp{v) be the conjugate 
of H by exp(t'). Then H* satisfies the conditions imposed on H (note that H*, like H, 
is compatible with A (see 10.8)), so there exist A* G C and c* G R, such that x^. = 
Im(A*77;j*) + c*yz* for all z* G 0^. Given z* G 0^ with y^* = 0, let z — e-xp{v)zex-p{—v). 
Because yz* — 0, we have [[z*, —v], —v] — 0, so, from Remark 7.23 and (7.22), we see that 

Vz = Yz* = 0, 

Vz = Vz' - (p-vYz' = Vz*, 

and 

x-z =Xz* +2 lm{(f)_yr]^) = x^* + 2 lm{{-X/2)jj;) = Xz* + Im(A7;2) = x^* + x^. 
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Therefore 

= X;j* = Im(A*772*) + c*\/z* = Ini(A*772«). 

Since rjz* is arbitrary, this imphes A* = 0. Thus, by replacing H with if*, we may assume 
that A = 0. This means that = cx^ for a\\ z E'O^. 

From (vi) (and because F = C, so g = 2), we know that dimc)(, = 3 > 1, so there is some 
nonzero w e ?)[,, such that = 0. (So = cy^ = 0.) Then + x^^y^ = |?7^p > 0, so we 
see, from (iv), that I772P + y^zyz > for every nonzero z EVt,. Now, since 

dimDf, — 3 > 2 — dimna+2/3, 

there is some nonzero 2; G D;,, such that rjz — 0. We have 

< \r]z\^ + y^zYz = + cyl- 

Because y^ is pure imaginary, we know that < 0, so this imphes that c < 0. Thus, 
replacing if by a conjugate under a diagonal matrix, we may assume c— —1, as desired. 

Step 1.2. Setting u' = (Uq + xVa+f}) n u, we have u = u' + Since T — ker a (see i), we have 
. /3|r=(a + /3)|T, 

• 2(5\t = (a + 2/3) It = (2q; + 2/3)|t, and 

• P\t ^ 2f3\T. 

Thus, in the notation of Lemma 3.21, we have n^^ fl u = u' and xv^^ fl u = c)(,, so u = u' © Of,, 
as desired. (Note that this is a direct sum of vector spaces, not of Lie algebras: we have 
[u',u'] C C)f,.) 

Step 1.3. Completion of the proof of Case 1. For any m e u' with Xu — 0, we have 

dimF(Fa;„ + Fy„) = dimpFy^ < 1 < 2, 

so M e u' n Of, = {0} (see iii); therefore. Lemma 8.26(1) implies there is a M-linear map 
B : F"~^ F"~^, such that = x^B for all u e u'. Then, because 

dimUa+/3 = dimjRF""^ = q{n — 2) = dimu' 

(see v), we must have 

u' = { e n/3 + ria+fs \ Vu = XuB }. 
Combining this with (i) and the conclusions of Steps 1.1 and 1.2, we see that \) — \}b- 
Therefore H — Hb, so Conclusion (Ic) holds. 

From (iii), we see that Conclusion (lb) holds. 

Letting z = [u,v], for any u,v & u', we see, from (7.22), that 

x^ = -2 lm{xuxl) 

and 

y, = -2lm{y^yl) = -2lm{{x^B){x,By) . 
From Step 1.1, wc know that yz = — x^, so this implies that Conclusion (la) holds. 

Case 2. Assume there does not exist a continuous curve /?/ 00 in H . such that p{h^) x 
Since dim if > qn, Theorem 10.26 implies that dimii = qn, and that H is of the 
form H — T \K U, where 

i) T = ker /3, 

ii) u = ((n„ + xia+p + na+2/3) n u) + n2a+2/3, and 

iii) + 2 Re{4>Jh) 7^ for every m e u \ n2a+2/3- 
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Let 

u' = {ria + ria+p + na+2p) n u 
(so u = u'©n2Q+2/3)- Let Q be the sesquilinear form (or bilinear form, if F = R) on F®F"'"^®F 
defined by 

Q((0i, xi, r]i), (02, X2, 772)) = (pim + a^i4 + ^102- 

Let 

Vt,^{{(f)u,xu,vu) eFeF^-'eF I liGu'}. 

Prom (iii), we see that the restriction of ReQ to Vj, is a (positive-definite) inner product. 

Let be the (Re Q) -orthogonal complement to Vf,. As a form over F, Q has signature 
(1, n — 1). Thus, as a form over M, ReQ has signature (g, q{n — 1)) . Since 

dim = dim i) — dim t — dim n2a+2i3 — — 1 — {q — 1) — q{n — 1), 

we conclude that Vf^- is a g-dimensional M-subspace on which Re Q is negative-definite. 

Choose some nonzero u G V^^-. Multiplying by a real scalar to normalize, we may assume 
Q{u,u) = —2. Because SU(l,n — 1) is transitive on the vectors of norm —1, there is some 
g e SU(ReQ), such that g{u) = (1,0, -1). Thus, letting 

A 0\ 

(11.3) ^= g eSU(2,n;F), 

\0 ij 

and f)!* = we have (1, 0, —1) e V(,j, so, by replacing f) with the conjugate f)!*, we may 

assume u — (1, 0, —1). 
Then 

V(, C 

= (1,0,-1)^ 

(11.4) = { (0, X, r/) I Re g((0, x, r]), (1, 0, -1)) = } 

= { (0, X, 7]) I Re(0(-1) + a;(Ot) + r){l)) = } 
= { (0, x,?7) I Rer^ = Re0}. 

Subcase 2.1. Assume F = R. By comparing (11.4) and (7.13) (with F = R), we conclude 
that 

[) C SU(1, n;M) n (a + n). 
By comparing dimensions, wc sec that equality must hold; this establishes Conclusion (2). 

Subcase 2.2. Assume F = C. Choose some nonzero v G V^, such that v is (Re Q) -orthogonal 
to u. Multiplying by a real scalar to normalize, we may assume Q{v,v) = —2. By replacing 
V with —V if necessary, we may assume (lmQ[u,v))/i > 0. 

Because v is (Re(5)-orthogonal to u — (1,0, —1), we have Re?7^ = Re0„ (see 11.4). Let 
s — (Im0„)/i and t — (Im?7„)/i. Then 

< {lmQ{u,v))/i 

= (lm((l)r^; + O(4) + (-l)0:))A 

= (-Im77„ + Im0^)/i 
^-t + s, 
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SO 

(lm.Q{u,v))/i — \s — t\. 

Also, 

-2^Q{v,v) 

= + 2Re(0^77;;) 

= |x^|^ + 2(Re(/)^)^-2(Im0^)(Im77^) 

> -2(Im0^)(Im?7^) 

= 2st, 

so st < —1. Thus, s and t are of opposite signs so, because |s| \t\ > 1, we have 

{lmQ{u,v))/i =\s-t\ = \s\ + \t\ > 2. 
Therefore, we may choose c e (0, 1], such that 

lmQ{u, v) — i ( c-\ — 
V c 

Let 

fi \ 
w — I -, 0, —zcj . 

Then 

Q{w,w) = \xyj\^ + 2Re{(j)^fj;;;) = O"^ + 2{i/c){ic) = -2 = Q{v,v), 



and 



Q{u,w) = (puVw + Xuxl + r}u(l)w = (l)(^c) + 0+ (-l)(-i/c) = i{c+-) = lmQ{u,v). 



c 



Hence, there is some h e S\J{Q), such that h{u) — u and h{v) — w. Thus, replacing f) with 
the conjugate h~^i)h (cf. 11.3), we may assume v — w. 
Therefore 

= (i,0,-icV 

= { (0, X, ri) I Re g((0, X, ri), (i, 0, -ic^)) = } 

= { (0, X, Tj) I Re(0(ic^) + x(Ot) + rj{-i)) = } 

= { (0, X, 77) I Im ?7 = Im }. 

By combining this with (11.4) and comparing with (9.11) (with F = C), we conclude that 
I) C {)[c2]. By comparing dimensions, we see that equality must hold; this establishes Con- 
clusion (3) (because < < 1). □ 

Theorem 11.2" can be restated in the following more elementary (but less precise) form. 

11.5. Corollary. Let H be a closed, connected subgroup of G — SU(2,2m;F) with m>2, 
such that neither H nor G/H is compact, and let q = dim^F. 

The homogeneous space G/H has a tessellation if and only if 

1) d{H) = 2qm; and 
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2) either H r.S\J{l,2m;¥) or ~ Sp(l, m; F). 

Proof. (^) This is Theorem 9.2. 

(^) Theorem 11.2(=^) provides us with three possibihties. 

(1) In each case, we have d{H) — 2qm (sec 9.5, 7.15, and 9.12). 

(2) In each case, there is some k G {1,2}, such that p{h) x \\h\\^ for h & H (see 9.7(1), 
8.21(1), and 9.14(1)). Then Corollary 8.23 implies either that H SU(l,2m;F) (if k = 1) 
or that H ~ Sp(l, m; F) (if A; = 2). □ 

The following proposition shows that no further restriction can be placed on c in the 
statement of Theorem 11.2(3). 

11.6. Proposition. If¥ — C, then H^c] is not conjugate to i/[c'], unless c — d {for c,c' e 
(0,1]). 

Proof. Suppose g~^H[c]g — H[c'], for some g E G — SU(2,2m). Because all maximal split 
tori in H[c'] are conjugate, we may assume that g normalizes ker (3. Since all roots of ker /? 
on both and f)[c'] are positive, g cannot invert ker/?, so we conclude that g centralizes 
ker/3; that is, g e CG(ker/3). 

In the notation of Case 2 of the proof of Theorem 11.2, define 

S={h\he SU(g)} 

(cf. 11.3). Then CG(ker/3) = (ker/3)5', so we may assume g E S (because ker/3, being a 
subgroup of H[c], obviously normalizes i?[c])- Write g — h. Then, because g''^H[c]g — -f^[c'], 
we must have /i(V(,(^,) = Vi,^^,^; hence h{Vf^^^) = 

For any basis {u, v} of V^^^^ with Q{u, u) — Q{v, v) — —2 and ReQ{u, v) — 0, we have 

lmQ{u,v) = ±i{c+ (1/c)). 

Similarly for any (Re(5)-orthonormal basis {u',v'} of Because h G SU((5), this implies 

c + (1/c) = c' + (1/c'). Because c, c' e (0, 1], we conclude that c = c', as desired. □ 
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